2024-03-25T16:10:30.299 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:10:30.299 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/orig.config.yaml 2024-03-25T16:10:30.344 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:10:30.440 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621254' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmDuJIdzytL+sQK2u890CdR23URp81hjZiYk0h3GOYGZrh4ReOTPR/CTG9+Yf3U+ctPKclt7vHrQJeTXjsoR1A= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/wg/nRE65EJryN827OMAbsAXfPhKVsgN7uskCi6+GuQK9UkPx1JoN6arIrnJiWmNxcKP2nzDLkXe0PoG+Tv8Q= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:10:30.440 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:10:30.441 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:10:30.441 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:10:30.442 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:10:30.468 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:10:30.468 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:10:30.468 INFO:teuthology.packaging:ref: None 2024-03-25T16:10:30.468 INFO:teuthology.packaging:tag: None 2024-03-25T16:10:30.468 INFO:teuthology.packaging:branch: main 2024-03-25T16:10:30.468 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:10:30.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:10:30.889 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:10:30.891 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:10:30.897 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:10:30.897 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:10:30.947 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:10:30.964 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:10:30.970 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:10:30.994 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:25.553767', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmDuJIdzytL+sQK2u890CdR23URp81hjZiYk0h3GOYGZrh4ReOTPR/CTG9+Yf3U+ctPKclt7vHrQJeTXjsoR1A='} 2024-03-25T16:10:31.016 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:25.551908', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/wg/nRE65EJryN827OMAbsAXfPhKVsgN7uskCi6+GuQK9UkPx1JoN6arIrnJiWmNxcKP2nzDLkXe0PoG+Tv8Q='} 2024-03-25T16:10:31.016 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:10:31.022 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:10:31.022 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:10:31.023 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:10:31.117 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f376f8ed0d0>, signals=[15]) 2024-03-25T16:10:31.117 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:10:31.128 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:10:31.129 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T16:10:31.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:31.231 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T16:10:31.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:10:31.322 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:10:31.329 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-03-25T16:10:31.347 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-03-25T16:10:31.347 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:10:31.404 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:10:31.405 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:10:31.405 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:10:31.405 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:10:31.405 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:10:31.405 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-03-25T16:10:31.429 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-03-25T16:10:31.447 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-03-25T16:10:31.448 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-03-25T16:10:31.503 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-03-25T16:10:31.503 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-03-25T16:10:31.503 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-03-25T16:10:31.503 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:10:31.503 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-03-25T16:10:31.503 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:10:31.504 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:10:31.504 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-03-25T16:10:31.504 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:10:31.504 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:10:31.504 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:10:31.504 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:10:31.504 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:10:31.504 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:10:31.504 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-03-25T16:10:31.528 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:10:31.540 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:10:31.562 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:10:31.562 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:10:31.564 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:10:31.581 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:10:31.586 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:10:31.586 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:10:31.621 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:10:31.650 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:10:31.704 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:10:31.720 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:10:31.720 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:10:31.720 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:10:31.721 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:10:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:31.737 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-25T16:10:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:10:31.738 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-25T16:10:31.793 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:10:31.794 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-361.el9.x86_64 2024-03-25T16:10:31.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-03-25T16:10:31.795 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:10:31.795 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-361.el9.x86_64 2024-03-25T16:10:31.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-03-25T16:10:33.879 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:10:33.994 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:10:35.538 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-25T16:10:35.645 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-25T16:10:36.485 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-25T16:10:36.562 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T16:10:38.294 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-25T16:10:42.112 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-25T16:10:43.587 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-25T16:10:47.404 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 21 MB 00:03 2024-03-25T16:10:52.412 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:10:54.527 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:10:54.527 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:10:54.662 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:10:54.674 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:10:54.675 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:10:54.676 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:10:54.676 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-03-25T16:10:54.676 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:10:54.676 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 76 M 2024-03-25T16:10:54.676 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 123 M 2024-03-25T16:10:54.676 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:10:55.196 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 119 kB/s | 44 kB 00:00 2024-03-25T16:10:56.830 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.4 MB/s | 15 MB 00:02 2024-03-25T16:10:59.721 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 6.8 MB/s | 33 MB 00:04 2024-03-25T16:11:01.984 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 791 kB/s | 19 MB 00:24 2024-03-25T16:11:06.661 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 2.4 MB/s | 27 MB 00:11 2024-03-25T16:11:06.662 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:06.662 INFO:teuthology.orchestra.run.smithi162.stdout:Total 6.3 MB/s | 76 MB 00:11 2024-03-25T16:11:07.324 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:11:07.709 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:11:07.709 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:11:07.785 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.1 MB 00:02 2024-03-25T16:11:08.735 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:11:08.735 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:11:09.527 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-25T16:11:11.208 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:11:11.815 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:11.902 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:14.795 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:16.322 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 3.1 MB/s | 21 MB 00:06 2024-03-25T16:11:19.548 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:19.652 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:21.236 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-25T16:11:23.318 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:11:23.318 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:11:23.418 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:11:23.422 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-03-25T16:11:23.423 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:11:23.424 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 76 M 2024-03-25T16:11:23.424 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 123 M 2024-03-25T16:11:23.424 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:11:23.995 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 168 kB/s | 44 kB 00:00 2024-03-25T16:11:24.450 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:25.237 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-modules-core-5.14.0-430.el9.x86_6 22 MB/s | 27 MB 00:01 2024-03-25T16:11:25.829 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.1 MB/s | 15 MB 00:02 2024-03-25T16:11:28.945 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 6.4 MB/s | 33 MB 00:05 2024-03-25T16:11:28.948 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:11:28.948 INFO:teuthology.orchestra.run.smithi191.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-25T16:11:29.608 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:11:29.955 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:11:29.955 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:11:30.965 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:11:30.965 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:11:33.441 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:11:34.043 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:34.163 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:36.900 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:41.776 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:41.825 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:46.544 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:53.673 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:53.674 INFO:teuthology.orchestra.run.smithi162.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:11:53.674 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:11:53.718 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:54.090 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:54.090 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:11:54.091 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:11:54.091 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:11:54.501 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:11:54.501 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:11:54.501 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:11:54.501 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:54.501 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:54.501 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:11:54.501 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:11:54.501 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:11:54.502 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:11:54.807 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:54.948 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:54.948 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:54.948 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:11:54.948 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:11:54.948 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:11:54.948 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:11:54.949 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:11:54.949 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:11:55.028 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:11:55.029 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:11:55.029 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-03-25T16:11:55.064 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-03-25T16:11:55.064 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-03-25T16:11:55.064 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2024-03-25T16:11:55.064 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 68.el9 2024-03-25T16:11:55.064 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-03-25T16:11:55.065 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:11:55.065 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2024-03-25T16:11:55.065 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8264254 2024-03-25T16:11:55.065 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-03-25T16:11:55.065 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:11:55.065 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:11:55.066 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:11:55.066 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:11:55.066 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2024-03-25T16:11:55.066 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-03-25T16:11:55.066 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:11:55.066 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-03-25T16:11:55.067 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-03-25T16:11:55.067 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:11:55.067 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:11:55.067 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:11:55.067 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:11:55.068 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-03-25T16:11:55.068 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:11:55.068 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:11:55.069 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:11:55.069 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:11:55.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:11:55.709 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-03-25T16:11:57.139 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:11:57.160 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-03-25T16:11:57.162 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-25T16:11:57.177 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.QWX4L8UZ3w 2024-03-25T16:11:57.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QWX4L8UZ3w 2024-03-25T16:11:57.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.QWX4L8UZ3w 2024-03-25T16:11:57.387 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.QWX4L8UZ3w is 6KB 2024-03-25T16:11:57.436 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.QWX4L8UZ3w 2024-03-25T16:11:57.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:11:57.516 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:11:57.516 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:11:57.516 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:11:57.516 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:11:57.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:11:57.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-03-25T16:12:16.014 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:16.014 INFO:teuthology.orchestra.run.smithi191.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:12:16.014 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:12:16.057 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:16.419 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:16.419 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:12:16.419 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:12:16.419 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:12:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:12:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:12:16.834 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:12:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:12:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:12:16.834 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:12:16.835 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:12:17.153 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:17.265 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:17.265 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:17.265 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:12:17.265 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:12:17.265 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:12:17.265 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:12:17.265 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:12:17.266 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:12:17.345 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:12:17.345 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:12:17.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-03-25T16:12:17.407 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 68.el9 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8264254 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:12:17.408 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:12:17.409 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:12:17.410 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:12:17.410 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:12:17.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:12:17.985 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-03-25T16:12:19.415 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:12:19.444 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-03-25T16:12:19.446 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-25T16:12:19.461 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.vJN3hi7TfT 2024-03-25T16:12:19.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vJN3hi7TfT 2024-03-25T16:12:19.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.vJN3hi7TfT 2024-03-25T16:12:19.661 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.vJN3hi7TfT is 6KB 2024-03-25T16:12:19.710 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.vJN3hi7TfT 2024-03-25T16:12:19.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:12:19.798 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:12:19.798 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:12:19.798 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:12:19.798 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:12:19.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:12:19.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-03-25T16:12:27.674 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:12:27.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-25T16:12:27.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:46.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-25T16:12:49.946 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:12:49.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-25T16:12:49.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:55.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-25T16:12:55.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:12:58.306 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-25T16:13:08.510 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-03-25T16:13:10.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-25T16:13:10.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:17.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-25T16:13:17.516 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:20.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-03-25T16:13:28.738 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-25T16:13:32.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-25T16:13:32.579 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:43.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-25T16:13:43.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:13:44.140 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-25T16:13:44.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-25T16:13:44.574 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:13:44.574 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-25T16:13:44.588 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:13:44.589 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:13:44.589 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:13:44.589 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:13:45.589 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:13:45.590 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:13:45.672 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-03-25T16:13:45.717 DEBUG:teuthology.parallel:result is None 2024-03-25T16:14:04.392 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-25T16:14:04.968 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-25T16:14:04.968 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:14:04.969 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-03-25T16:14:04.992 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:14:04.992 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:14:04.992 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:14:04.992 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:14:05.992 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:14:05.993 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:14:06.073 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-03-25T16:14:06.110 DEBUG:teuthology.parallel:result is None 2024-03-25T16:14:06.111 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:14:06.120 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:14:06.120 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:14:06.124 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:14:06.140 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:14:06.156 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:14:06.190 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:14:06.191 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:14:06.193 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:14:06.577 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:14:06.592 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:14:06.592 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:14:06.601 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:14:06.629 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:06.644 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:06.666 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:06.681 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:14:06.682 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:14:06.697 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:14:06.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:14:06.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:14:06.729 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:14:06.743 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:14:06.743 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:14:06.771 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:14:06.786 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:06.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:06.925 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:07.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:07.050 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T16:14:07.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:14:07.117 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:07.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:14:07.208 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:07.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:14:07.326 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T16:14:07.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:14:07.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-03-25T16:14:07.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-03-25T16:14:07.441 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:14:07.483 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:14:07.812 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:14:07.820 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:14:07.820 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:14:07.861 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:14:07.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-03-25T16:14:07.974 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-03-25T16:14:07.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-03-25T16:14:08.036 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-03-25T16:14:08.039 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:14:08.039 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-03-25T16:14:08.058 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-03-25T16:14:08.058 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-03-25T16:14:08.087 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-03-25T16:14:08.088 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi162.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:14:08.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:14:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:08.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:14:08.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:08.153 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:14:08.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:14:08.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:14:08.238 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:14:08.248 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:14:08.266 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:14:08.268 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi162.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:18:21.680 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi162.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-03-25T16:18:21.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-25T16:18:21.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:21.772 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-25T16:18:21.848 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-25T16:18:21.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-25T16:18:21.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:21.937 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-03-25T16:18:22.020 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-03-25T16:18:22.021 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:18:22.035 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:18:22.036 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:18:22.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:18:22.038 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:18:22.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:18:22.066 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:18:22.083 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:18:22.091 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:18:22.108 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:18:22.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-03-25T16:18:22.126 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-03-25T16:18:22.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-03-25T16:18:22.142 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:18:22.150 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-03-25T16:18:22.159 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:18:22.167 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:18:22.184 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:18:22.217 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:18:22.220 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:18:22.220 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-25T16:18:22.220 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:22.220 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:22.221 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:22.221 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:22.242 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:18:22.245 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:18:22.245 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-25T16:18:22.245 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:22.245 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:22.245 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:22.245 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:18:22.246 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:18:22.254 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:18:22.254 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:18:22.254 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:18:22.255 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:18:22.260 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:18:22.260 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:18:22.261 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:18:22.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:18:22.263 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:18:22.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:18:22.483 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:18:22.483 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:18:22.487 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:18:22.487 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:18:22.818 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:18:22.819 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T16:18:22.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:18:22.848 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:18:22.848 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:18:22.879 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:18:22.880 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T16:18:22.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:18:22.911 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:18:22.911 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:18:22.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:18:22.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:18:22.998 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:18:23.064 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:18:23.065 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-03-25T16:18:23.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-03-25T16:18:23.132 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-03-25T16:18:23.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-03-25T16:18:23.306 INFO:teuthology.orchestra.run.smithi162.stdout:74 files removed 2024-03-25T16:18:23.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-03-25T16:18:23.375 INFO:teuthology.orchestra.run.smithi191.stdout:74 files removed 2024-03-25T16:18:23.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2024-03-25T16:18:24.594 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 85 kB/s | 82 kB 00:00 2024-03-25T16:18:24.791 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 75 kB/s | 82 kB 00:01 2024-03-25T16:18:25.344 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T16:18:25.583 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:18:26.186 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:18:26.318 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-25T16:18:26.596 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T16:18:26.754 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T16:18:28.354 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-25T16:18:29.564 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 1.1 MB/s | 2.0 MB 00:01 2024-03-25T16:18:30.495 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.0 MB 00:01 2024-03-25T16:18:33.172 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-25T16:18:33.253 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.0 MB 00:03 2024-03-25T16:18:36.054 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-25T16:18:37.081 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-25T16:18:38.674 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 86 kB/s | 16 kB 00:00 2024-03-25T16:18:39.795 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:01 2024-03-25T16:18:40.063 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-25T16:18:41.613 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-25T16:18:44.115 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 21 MB 00:02 2024-03-25T16:18:44.743 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:18:47.197 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:18:47.225 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options 2024-03-25T16:18:47.226 INFO:teuthology.orchestra.run.smithi191.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:18:47.227 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-03-25T16:18:47.228 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:18:47.229 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:18:47.229 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:18:47.229 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:18:47.229 INFO:teuthology.orchestra.run.smithi191.stdout:Install 30 Packages 2024-03-25T16:18:47.229 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2024-03-25T16:18:47.229 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:18:47.229 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 58 M 2024-03-25T16:18:47.230 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:18:49.147 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T16:18:51.584 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:18:51.591 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:18:51.591 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:18:51.592 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse 2024-03-25T16:18:51.593 INFO:teuthology.orchestra.run.smithi162.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:18:51.594 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:18:51.595 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:18:51.595 INFO:teuthology.orchestra.run.smithi162.stdout:Install 30 Packages 2024-03-25T16:18:51.595 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-03-25T16:18:51.595 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:18:51.595 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 58 M 2024-03-25T16:18:51.595 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:18:51.914 INFO:teuthology.orchestra.run.smithi191.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 5.3 MB 00:04 2024-03-25T16:18:52.019 INFO:teuthology.orchestra.run.smithi191.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 233 kB/s | 25 kB 00:00 2024-03-25T16:18:52.486 INFO:teuthology.orchestra.run.smithi191.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.6 MB/s | 742 kB 00:00 2024-03-25T16:18:53.037 INFO:teuthology.orchestra.run.smithi191.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 906 kB/s | 498 kB 00:00 2024-03-25T16:18:55.045 INFO:teuthology.orchestra.run.smithi191.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 9.7 MB 00:07 2024-03-25T16:18:55.146 INFO:teuthology.orchestra.run.smithi191.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 451 kB/s | 45 kB 00:00 2024-03-25T16:18:55.312 INFO:teuthology.orchestra.run.smithi191.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 789 kB/s | 131 kB 00:00 2024-03-25T16:18:55.505 INFO:teuthology.orchestra.run.smithi191.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 844 kB/s | 162 kB 00:00 2024-03-25T16:18:55.763 INFO:teuthology.orchestra.run.smithi191.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.2 MB/s | 320 kB 00:00 2024-03-25T16:18:56.089 INFO:teuthology.orchestra.run.smithi191.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 916 kB/s | 298 kB 00:00 2024-03-25T16:18:56.247 INFO:teuthology.orchestra.run.smithi191.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 627 kB/s | 99 kB 00:00 2024-03-25T16:18:56.356 INFO:teuthology.orchestra.run.smithi191.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-25T16:18:56.407 INFO:teuthology.orchestra.run.smithi191.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:18:56.440 INFO:teuthology.orchestra.run.smithi191.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:18:56.507 INFO:teuthology.orchestra.run.smithi191.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:18:56.549 INFO:teuthology.orchestra.run.smithi191.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:18:56.583 INFO:teuthology.orchestra.run.smithi191.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:18:57.075 INFO:teuthology.orchestra.run.smithi191.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 440 kB/s | 216 kB 00:00 2024-03-25T16:18:57.342 INFO:teuthology.orchestra.run.smithi191.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 131 kB/s | 35 kB 00:00 2024-03-25T16:18:57.868 INFO:teuthology.orchestra.run.smithi191.stdout:(20/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 2.0 MB/s | 20 MB 00:09 2024-03-25T16:18:57.893 INFO:teuthology.orchestra.run.smithi191.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 81 kB/s | 44 kB 00:00 2024-03-25T16:18:57.997 INFO:teuthology.orchestra.run.smithi162.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.6 MB/s | 9.7 MB 00:06 2024-03-25T16:18:58.108 INFO:teuthology.orchestra.run.smithi162.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 228 kB/s | 25 kB 00:00 2024-03-25T16:18:58.518 INFO:teuthology.orchestra.run.smithi191.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 475 kB/s | 309 kB 00:00 2024-03-25T16:18:58.619 INFO:teuthology.orchestra.run.smithi191.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-03-25T16:18:58.658 INFO:teuthology.orchestra.run.smithi162.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 742 kB 00:00 2024-03-25T16:18:58.719 INFO:teuthology.orchestra.run.smithi191.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-25T16:18:58.845 INFO:teuthology.orchestra.run.smithi191.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-03-25T16:18:58.903 INFO:teuthology.orchestra.run.smithi191.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2024-03-25T16:18:59.012 INFO:teuthology.orchestra.run.smithi191.stdout:(27/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 874 kB/s | 5.1 MB 00:05 2024-03-25T16:18:59.038 INFO:teuthology.orchestra.run.smithi191.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-25T16:18:59.050 INFO:teuthology.orchestra.run.smithi162.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.2 MB/s | 498 kB 00:00 2024-03-25T16:18:59.121 INFO:teuthology.orchestra.run.smithi191.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-03-25T16:18:59.200 INFO:teuthology.orchestra.run.smithi162.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 736 kB/s | 5.3 MB 00:07 2024-03-25T16:18:59.300 INFO:teuthology.orchestra.run.smithi162.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-25T16:18:59.476 INFO:teuthology.orchestra.run.smithi162.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 751 kB/s | 131 kB 00:00 2024-03-25T16:18:59.817 INFO:teuthology.orchestra.run.smithi162.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 474 kB/s | 162 kB 00:00 2024-03-25T16:18:59.847 INFO:teuthology.orchestra.run.smithi191.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-03-25T16:19:00.359 INFO:teuthology.orchestra.run.smithi162.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 592 kB/s | 320 kB 00:00 2024-03-25T16:19:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 344 kB/s | 298 kB 00:00 2024-03-25T16:19:01.463 INFO:teuthology.orchestra.run.smithi191.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 3.1 MB 00:02 2024-03-25T16:19:01.710 INFO:teuthology.orchestra.run.smithi162.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 206 kB/s | 99 kB 00:00 2024-03-25T16:19:01.810 INFO:teuthology.orchestra.run.smithi162.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-25T16:19:01.860 INFO:teuthology.orchestra.run.smithi162.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:19:01.894 INFO:teuthology.orchestra.run.smithi162.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:19:01.952 INFO:teuthology.orchestra.run.smithi162.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:19:01.994 INFO:teuthology.orchestra.run.smithi162.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:19:02.036 INFO:teuthology.orchestra.run.smithi162.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2024-03-25T16:19:02.488 INFO:teuthology.orchestra.run.smithi191.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 3.4 MB 00:03 2024-03-25T16:19:02.495 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:02.495 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.8 MB/s | 58 MB 00:15 2024-03-25T16:19:02.527 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:19:02.564 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:19:02.564 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:19:02.645 INFO:teuthology.orchestra.run.smithi162.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 356 kB/s | 216 kB 00:00 2024-03-25T16:19:02.770 INFO:teuthology.orchestra.run.smithi162.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 279 kB/s | 35 kB 00:00 2024-03-25T16:19:02.870 INFO:teuthology.orchestra.run.smithi162.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 445 kB/s | 44 kB 00:00 2024-03-25T16:19:03.030 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:19:03.031 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:19:03.212 INFO:teuthology.orchestra.run.smithi162.stdout:(21/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 5.1 MB 00:04 2024-03-25T16:19:03.337 INFO:teuthology.orchestra.run.smithi162.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 662 kB/s | 309 kB 00:00 2024-03-25T16:19:03.429 INFO:teuthology.orchestra.run.smithi162.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-03-25T16:19:03.530 INFO:teuthology.orchestra.run.smithi162.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-03-25T16:19:03.621 INFO:teuthology.orchestra.run.smithi162.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 736 kB/s | 67 kB 00:00 2024-03-25T16:19:03.797 INFO:teuthology.orchestra.run.smithi162.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-25T16:19:03.897 INFO:teuthology.orchestra.run.smithi162.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-25T16:19:04.014 INFO:teuthology.orchestra.run.smithi162.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-03-25T16:19:04.081 INFO:teuthology.orchestra.run.smithi162.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-25T16:19:04.470 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:19:04.595 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:19:04.702 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:19:04.728 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:19:04.751 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:19:04.768 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:19:04.964 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:19:05.024 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:19:05.348 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:19:05.406 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:05.490 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:05.580 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:19:05.609 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:19:05.655 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:19:05.687 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:19:05.728 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:19:05.797 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:19:05.821 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:19:05.883 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:19:05.911 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:19:06.022 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:19:06.062 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:19:06.103 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:19:06.136 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:19:06.165 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:19:06.207 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:19:06.557 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:19:06.587 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:19:06.677 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:19:06.700 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:19:06.742 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:19:07.206 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:19:07.232 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:19:07.304 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:19:07.371 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:19:07.402 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:19:08.172 INFO:teuthology.orchestra.run.smithi162.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 783 kB/s | 3.1 MB 00:04 2024-03-25T16:19:09.533 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:09.584 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:10.129 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:10.363 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:19:10.439 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:19:10.439 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:19:10.439 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:19:10.440 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:19:10.474 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:10.781 INFO:teuthology.orchestra.run.smithi162.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 511 kB/s | 3.4 MB 00:06 2024-03-25T16:19:12.131 INFO:teuthology.orchestra.run.smithi162.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 20 MB 00:20 2024-03-25T16:19:12.135 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:19:12.135 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.8 MB/s | 58 MB 00:20 2024-03-25T16:19:12.170 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:19:12.207 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:19:12.207 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:19:12.672 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:19:12.673 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:19:14.164 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:19:14.298 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:19:14.336 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:19:14.361 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:19:14.384 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:19:14.401 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:19:14.598 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:19:14.649 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:19:15.026 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:19:15.062 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:15.154 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:15.241 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:19:15.272 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:19:15.335 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:19:15.363 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:19:15.424 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:19:15.491 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:19:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:19:15.562 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:19:15.589 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:19:15.699 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:19:15.733 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:19:15.773 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:19:15.805 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:19:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:19:15.867 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:19:16.176 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:19:16.205 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:19:16.295 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:19:16.326 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:19:16.372 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:19:16.825 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:19:16.858 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:19:16.912 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:19:16.981 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:19:17.010 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:19:19.208 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:19.256 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-03-25T16:19:19.672 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:19:19.814 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:19:19.943 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:19:20.026 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:19:20.026 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:19:20.026 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:19:20.026 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:19:20.069 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:20.588 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:19:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:19:20.612 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:19:20.612 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:19:20.612 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:19:20.612 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:19:20.612 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:19:20.614 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:19:20.699 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:19:20.700 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:21.845 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:21.845 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:19:21.845 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:19:21.846 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:19:21.847 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:19:21.848 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:19:21.848 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:19:21.848 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:19:21.848 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:19:21.848 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:19:21.849 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:21.849 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:19:21.849 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:19:22.833 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:22.833 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:19:22.833 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-03-25T16:19:22.833 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.833 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.833 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:19:22.834 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:19:22.835 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:19:22.836 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:19:23.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2024-03-25T16:19:23.703 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:39 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:19:23.859 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:19:23.860 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:19:23.860 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:19:23.860 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:19:23.860 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:19:23.860 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-03-25T16:19:23.861 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:19:23.862 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 48 M 2024-03-25T16:19:23.862 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 204 M 2024-03-25T16:19:23.862 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:19:24.069 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T16:19:24.111 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-25T16:19:24.136 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T16:19:24.687 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 214 kB/s | 163 kB 00:00 2024-03-25T16:19:24.970 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 224 kB/s | 187 kB 00:00 2024-03-25T16:19:29.285 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:29.285 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-03-25T16:19:29.285 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-03-25T16:19:29.286 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-03-25T16:19:29.286 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-03-25T16:19:29.286 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-03-25T16:19:29.286 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-03-25T16:19:29.286 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-03-25T16:19:29.286 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:19:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:19:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:19:30.218 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:19:30.218 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:19:30.218 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:19:30.218 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:19:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:19:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:19:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:19:30.304 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:31.445 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:31.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:19:31.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:19:31.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:19:31.445 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:19:31.446 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:19:31.446 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:19:31.446 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:19:31.447 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:19:31.447 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:19:31.447 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:19:31.447 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:19:31.447 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:19:31.448 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:19:31.449 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:19:32.462 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:19:32.462 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:19:32.463 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.464 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:19:32.465 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:19:32.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-03-25T16:19:33.285 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:44 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:19:33.438 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:19:33.439 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:19:33.440 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:19:33.440 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:19:33.440 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:19:33.440 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:19:33.440 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:19:33.440 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-03-25T16:19:33.440 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:19:33.440 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 48 M 2024-03-25T16:19:33.441 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 204 M 2024-03-25T16:19:33.441 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:19:33.842 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T16:19:33.967 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.8 MB/s | 300 kB 00:00 2024-03-25T16:19:34.017 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-25T16:19:34.317 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 626 kB/s | 187 kB 00:00 2024-03-25T16:19:34.901 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 137 kB/s | 163 kB 00:01 2024-03-25T16:20:28.754 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 747 kB/s | 47 MB 01:04 2024-03-25T16:20:28.756 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:28.756 INFO:teuthology.orchestra.run.smithi191.stdout:Total 760 kB/s | 48 MB 01:04 2024-03-25T16:20:28.769 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:20:28.789 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:20:28.789 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:20:29.049 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 875 kB/s | 47 MB 00:55 2024-03-25T16:20:29.051 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:29.051 INFO:teuthology.orchestra.run.smithi162.stdout:Total 887 kB/s | 48 MB 00:55 2024-03-25T16:20:29.063 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:20:29.083 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:20:29.083 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:20:29.175 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:20:29.176 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:20:29.461 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:20:29.462 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:20:29.772 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:20:29.845 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:20:29.923 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:20:29.963 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:20:30.013 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:20:30.039 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:20:30.048 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:20:30.115 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:20:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:20:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:20:30.240 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:20:30.269 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:20:34.488 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:20:34.587 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:20:34.901 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:20:34.978 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:20:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:20:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:20:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:20:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:20:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:20:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:20:37.348 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:20:37.348 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:20:37.348 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:20:37.349 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:20:37.349 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:20:37.349 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:20:37.357 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:20:37.357 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:20:37.357 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:20:37.358 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:37.358 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:20:37.358 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:37.358 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:20:37.358 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:20:37.358 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:20:37.358 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:20:37.358 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:20:37.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2024-03-25T16:20:37.689 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:20:37.689 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:20:37.689 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:20:37.689 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:37.690 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:20:37.690 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:20:37.690 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:20:37.690 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:20:37.690 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:20:37.690 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:20:37.690 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:20:38.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-03-25T16:20:38.218 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:20:38.374 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:20:38.377 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:20:38.378 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:20:38.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout:========================================================================================= 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout:Install 36 Packages 2024-03-25T16:20:38.380 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:20:38.381 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 30 M 2024-03-25T16:20:38.381 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 107 M 2024-03-25T16:20:38.381 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:20:38.576 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:49 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:20:38.736 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:20:38.738 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2024-03-25T16:20:38.738 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:20:38.739 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:20:38.740 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:20:38.741 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2024-03-25T16:20:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:Install 36 Packages 2024-03-25T16:20:38.742 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:20:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 30 M 2024-03-25T16:20:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 107 M 2024-03-25T16:20:38.742 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:20:39.355 INFO:teuthology.orchestra.run.smithi191.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:20:39.760 INFO:teuthology.orchestra.run.smithi162.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T16:20:41.202 INFO:teuthology.orchestra.run.smithi162.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 802 kB/s | 1.5 MB 00:01 2024-03-25T16:20:41.935 INFO:teuthology.orchestra.run.smithi162.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 825 kB/s | 2.1 MB 00:02 2024-03-25T16:20:42.496 INFO:teuthology.orchestra.run.smithi191.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 426 kB/s | 1.5 MB 00:03 2024-03-25T16:20:42.527 INFO:teuthology.orchestra.run.smithi162.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 418 kB/s | 247 kB 00:00 2024-03-25T16:20:42.644 INFO:teuthology.orchestra.run.smithi162.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:20:42.686 INFO:teuthology.orchestra.run.smithi162.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:20:42.719 INFO:teuthology.orchestra.run.smithi162.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-25T16:20:42.753 INFO:teuthology.orchestra.run.smithi162.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:20:42.787 INFO:teuthology.orchestra.run.smithi162.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:20:42.828 INFO:teuthology.orchestra.run.smithi162.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:20:42.904 INFO:teuthology.orchestra.run.smithi162.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:20:42.937 INFO:teuthology.orchestra.run.smithi162.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:20:42.992 INFO:teuthology.orchestra.run.smithi162.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:20:43.005 INFO:teuthology.orchestra.run.smithi162.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:20:43.163 INFO:teuthology.orchestra.run.smithi162.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 789 kB/s | 125 kB 00:00 2024-03-25T16:20:43.297 INFO:teuthology.orchestra.run.smithi162.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-03-25T16:20:43.497 INFO:teuthology.orchestra.run.smithi191.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 475 kB/s | 2.1 MB 00:04 2024-03-25T16:20:43.505 INFO:teuthology.orchestra.run.smithi162.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 726 kB/s | 151 kB 00:00 2024-03-25T16:20:43.564 INFO:teuthology.orchestra.run.smithi162.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 746 kB/s | 43 kB 00:00 2024-03-25T16:20:43.648 INFO:teuthology.orchestra.run.smithi162.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T16:20:43.748 INFO:teuthology.orchestra.run.smithi162.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T16:20:43.798 INFO:teuthology.orchestra.run.smithi162.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-03-25T16:20:43.848 INFO:teuthology.orchestra.run.smithi162.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-03-25T16:20:43.899 INFO:teuthology.orchestra.run.smithi162.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-03-25T16:20:43.949 INFO:teuthology.orchestra.run.smithi162.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-25T16:20:43.999 INFO:teuthology.orchestra.run.smithi162.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 394 kB/s | 20 kB 00:00 2024-03-25T16:20:44.050 INFO:teuthology.orchestra.run.smithi162.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 927 kB/s | 46 kB 00:00 2024-03-25T16:20:44.109 INFO:teuthology.orchestra.run.smithi162.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T16:20:44.138 INFO:teuthology.orchestra.run.smithi191.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 386 kB/s | 247 kB 00:00 2024-03-25T16:20:44.175 INFO:teuthology.orchestra.run.smithi162.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T16:20:44.226 INFO:teuthology.orchestra.run.smithi162.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-03-25T16:20:44.255 INFO:teuthology.orchestra.run.smithi191.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:20:44.284 INFO:teuthology.orchestra.run.smithi162.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T16:20:44.297 INFO:teuthology.orchestra.run.smithi191.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:20:44.331 INFO:teuthology.orchestra.run.smithi191.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-25T16:20:44.335 INFO:teuthology.orchestra.run.smithi162.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-03-25T16:20:44.356 INFO:teuthology.orchestra.run.smithi191.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:20:44.398 INFO:teuthology.orchestra.run.smithi191.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 927 kB/s | 39 kB 00:00 2024-03-25T16:20:44.401 INFO:teuthology.orchestra.run.smithi162.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T16:20:44.440 INFO:teuthology.orchestra.run.smithi191.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:20:44.477 INFO:teuthology.orchestra.run.smithi162.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T16:20:44.507 INFO:teuthology.orchestra.run.smithi191.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:20:44.527 INFO:teuthology.orchestra.run.smithi162.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-03-25T16:20:44.540 INFO:teuthology.orchestra.run.smithi191.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:20:44.566 INFO:teuthology.orchestra.run.smithi191.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 4.8 MB/s | 124 kB 00:00 2024-03-25T16:20:44.607 INFO:teuthology.orchestra.run.smithi191.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:20:45.033 INFO:teuthology.orchestra.run.smithi191.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 294 kB/s | 125 kB 00:00 2024-03-25T16:20:45.141 INFO:teuthology.orchestra.run.smithi191.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-03-25T16:20:45.483 INFO:teuthology.orchestra.run.smithi191.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 442 kB/s | 151 kB 00:00 2024-03-25T16:20:45.567 INFO:teuthology.orchestra.run.smithi191.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 522 kB/s | 43 kB 00:00 2024-03-25T16:20:45.675 INFO:teuthology.orchestra.run.smithi191.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-03-25T16:20:45.801 INFO:teuthology.orchestra.run.smithi191.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-25T16:20:45.861 INFO:teuthology.orchestra.run.smithi162.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 786 kB/s | 4.7 MB 00:06 2024-03-25T16:20:45.876 INFO:teuthology.orchestra.run.smithi191.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-03-25T16:20:45.943 INFO:teuthology.orchestra.run.smithi191.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 266 kB/s | 18 kB 00:00 2024-03-25T16:20:46.010 INFO:teuthology.orchestra.run.smithi191.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T16:20:46.077 INFO:teuthology.orchestra.run.smithi191.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 292 kB/s | 19 kB 00:00 2024-03-25T16:20:46.144 INFO:teuthology.orchestra.run.smithi191.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T16:20:46.227 INFO:teuthology.orchestra.run.smithi191.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 556 kB/s | 46 kB 00:00 2024-03-25T16:20:46.311 INFO:teuthology.orchestra.run.smithi191.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 944 kB/s | 79 kB 00:00 2024-03-25T16:20:46.403 INFO:teuthology.orchestra.run.smithi191.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-25T16:20:46.487 INFO:teuthology.orchestra.run.smithi191.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-03-25T16:20:46.603 INFO:teuthology.orchestra.run.smithi191.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 772 kB/s | 90 kB 00:00 2024-03-25T16:20:46.687 INFO:teuthology.orchestra.run.smithi191.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 430 kB/s | 36 kB 00:00 2024-03-25T16:20:46.787 INFO:teuthology.orchestra.run.smithi191.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-25T16:20:46.879 INFO:teuthology.orchestra.run.smithi191.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-03-25T16:20:46.963 INFO:teuthology.orchestra.run.smithi191.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-03-25T16:20:50.271 INFO:teuthology.orchestra.run.smithi191.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 439 kB/s | 4.7 MB 00:10 2024-03-25T16:20:57.692 INFO:teuthology.orchestra.run.smithi162.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 17 MB 00:16 2024-03-25T16:20:57.696 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:20:57.696 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.6 MB/s | 30 MB 00:18 2024-03-25T16:20:57.721 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:20:57.764 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:20:57.764 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:20:58.177 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:20:58.178 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:20:59.491 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:20:59.552 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:20:59.577 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:20:59.604 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:20:59.674 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:20:59.711 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:20:59.744 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:20:59.835 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:20:59.914 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:20:59.957 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:21:00.029 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:21:00.083 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:21:00.130 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:21:00.178 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:21:00.234 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:21:00.426 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:21:00.476 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:21:00.506 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:21:00.541 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:21:00.571 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:21:00.647 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:21:00.681 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:21:00.715 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:21:00.763 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:21:00.798 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:21:00.833 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:21:00.922 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:21:00.964 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:21:01.036 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:21:01.193 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:21:01.307 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:21:01.464 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:21:01.494 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:21:01.522 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:21:01.522 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:01.522 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:21:01.522 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:21:01.522 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:21:01.523 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:02.954 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:21:02.980 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:21:02.980 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:02.980 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:21:02.980 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:21:02.980 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:21:02.980 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:03.421 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:21:03.444 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:21:03.444 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:03.445 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:21:03.445 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:21:03.445 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:21:03.445 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:21:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:21:03.678 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:03.678 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:21:03.678 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:21:03.678 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:21:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:03.745 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:21:05.538 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:21:05.538 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:21:05.538 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:21:05.538 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:21:05.539 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:21:05.540 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:21:05.541 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:21:05.541 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:21:05.541 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:21:05.541 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:21:05.541 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:21:05.541 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:21:05.541 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:21:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:21:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:06.425 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:21:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:06.425 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:21:06.426 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:21:06.427 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:21:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:21:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:21:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:21:06.428 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:06.428 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:21:06.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-03-25T16:21:07.386 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:18 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:21:07.483 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:07.536 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:21:07.537 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:21:07.537 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:21:07.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-03-25T16:21:08.168 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:19 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:21:08.317 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:21:08.318 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:21:08.318 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:21:08.318 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:21:08.318 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:21:08.318 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:21:08.318 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:08.319 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:21:08.319 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:21:08.319 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:21:08.319 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:08.319 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 747 k 2024-03-25T16:21:08.319 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 754 k 2024-03-25T16:21:08.319 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:21:09.152 INFO:teuthology.orchestra.run.smithi191.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 645 kB/s | 17 MB 00:26 2024-03-25T16:21:09.156 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:09.156 INFO:teuthology.orchestra.run.smithi191.stdout:Total 987 kB/s | 30 MB 00:30 2024-03-25T16:21:09.178 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:21:09.219 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:21:09.219 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:21:09.636 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:21:09.637 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:21:09.755 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 521 kB/s | 747 kB 00:01 2024-03-25T16:21:09.755 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:09.756 INFO:teuthology.orchestra.run.smithi162.stdout:Total 520 kB/s | 747 kB 00:01 2024-03-25T16:21:09.756 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:21:09.762 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:21:09.762 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:21:09.768 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:21:09.769 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:21:09.846 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:21:10.300 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:10.376 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:10.798 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:10.972 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:21:11.037 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:21:11.063 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:11.063 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:11.063 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:21:11.063 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:11.063 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:11.063 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:21:11.075 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:21:11.106 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:21:11.177 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:21:11.221 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:21:11.255 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:21:11.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:21:11.347 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:21:11.583 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:21:11.624 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:21:11.696 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:21:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:21:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:21:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:21:11.862 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:22 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:21:11.900 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:21:12.017 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:21:12.017 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 145 k 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 436 k 2024-03-25T16:21:12.018 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:21:12.084 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:21:12.134 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:21:12.164 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:21:12.198 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:21:12.228 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:21:12.304 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:21:12.338 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:21:12.372 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:21:12.412 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:21:12.447 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:21:12.490 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:21:12.579 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:21:12.629 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:21:12.702 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:21:12.803 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 185 kB/s | 145 kB 00:00 2024-03-25T16:21:12.803 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:12.803 INFO:teuthology.orchestra.run.smithi162.stdout:Total 185 kB/s | 145 kB 00:00 2024-03-25T16:21:12.804 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:21:12.812 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:21:12.812 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:21:12.846 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:21:12.866 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:21:12.867 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:21:12.992 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:21:13.008 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:21:13.070 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:13.095 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:13.095 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:13.095 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:21:13.095 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:13.155 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:21:13.184 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:21:13.208 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:21:13.209 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:13.209 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:21:13.209 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:21:13.209 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:21:13.209 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:14.083 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:14.084 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:14.084 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:21:14.084 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:14.084 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:14.084 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:21:14.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-03-25T16:21:14.644 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:21:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:21:14.670 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:14.670 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:21:14.670 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:21:14.670 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:21:14.670 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:14.900 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:25 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:21:14.998 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:15.050 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:21:15.051 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:21:15.051 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:21:15.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:21:15.137 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:21:15.164 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:21:15.164 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:15.164 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:21:15.164 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:21:15.164 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:21:15.164 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:15.371 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:21:15.399 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:21:15.400 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:15.400 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:21:15.400 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:21:15.400 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:21:15.400 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:15.461 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:21:15.683 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:02:26 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:21:15.832 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:21:15.833 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-03-25T16:21:15.834 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:21:15.835 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 6.9 M 2024-03-25T16:21:15.835 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2024-03-25T16:21:15.835 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:21:16.749 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 70 kB/s | 31 kB 00:00 2024-03-25T16:21:16.858 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 28 kB/s | 15 kB 00:00 2024-03-25T16:21:16.916 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-25T16:21:16.941 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:21:17.146 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:21:17.147 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:21:17.148 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:21:17.148 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:21:17.149 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:21:17.150 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:21:17.200 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-25T16:21:17.367 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T16:21:17.643 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.4 MB/s | 2.0 MB 00:00 2024-03-25T16:21:17.726 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-25T16:21:17.818 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 336 kB/s | 31 kB 00:00 2024-03-25T16:21:17.902 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T16:21:18.053 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:21:18.054 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:18.054 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:21:18.054 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:18.054 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:18.055 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:18.055 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:18.055 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:18.055 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:18.055 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:21:18.055 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:21:18.056 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:21:18.056 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:21:18.056 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:21:18.056 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:21:18.056 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:21:18.056 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:21:18.057 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:18.058 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:18.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2024-03-25T16:21:18.982 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:34 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:19.079 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:19.130 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:19.131 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:21:19.131 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:19.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2024-03-25T16:21:19.752 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:35 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:19.903 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 747 k 2024-03-25T16:21:19.904 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 754 k 2024-03-25T16:21:19.905 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:21:20.260 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 461 kB/s | 1.8 MB 00:03 2024-03-25T16:21:21.453 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 483 kB/s | 747 kB 00:01 2024-03-25T16:21:21.453 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:21.454 INFO:teuthology.orchestra.run.smithi191.stdout:Total 482 kB/s | 747 kB 00:01 2024-03-25T16:21:21.454 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:21:21.460 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:21:21.460 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:21:21.466 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:21:21.467 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:21:21.544 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:21:22.015 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:22.134 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:22.518 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:22.812 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:21:22.812 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:22.812 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:21:22.812 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:22.812 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:22.813 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:23.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:21:23.589 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:39 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:23.744 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:23.744 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:23.744 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:21:23.744 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:23.744 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:21:23.744 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:21:23.744 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:23.745 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:21:23.745 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:23.745 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:21:23.745 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:23.745 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 145 k 2024-03-25T16:21:23.745 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 436 k 2024-03-25T16:21:23.745 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:21:24.350 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 240 kB/s | 145 kB 00:00 2024-03-25T16:21:24.351 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:24.351 INFO:teuthology.orchestra.run.smithi191.stdout:Total 240 kB/s | 145 kB 00:00 2024-03-25T16:21:24.351 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:21:24.359 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:21:24.359 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:21:24.413 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:21:24.413 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:21:24.555 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:21:24.626 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:24.650 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:24.651 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:21:24.651 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:21:24.651 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:25.603 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:21:25.603 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:25.603 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:21:25.603 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:21:25.603 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:25.603 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:25.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2024-03-25T16:21:26.451 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:42 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:26.549 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:21:26.601 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:26.602 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:21:26.602 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:26.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:21:27.228 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:43 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:27.377 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:27.378 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:21:27.379 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout:====================================================================================== 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-03-25T16:21:27.380 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:27.381 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 6.9 M 2024-03-25T16:21:27.381 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 26 M 2024-03-25T16:21:27.381 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:21:28.071 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 71 kB/s | 31 kB 00:00 2024-03-25T16:21:28.229 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-25T16:21:28.271 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T16:21:28.330 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 22 kB/s | 15 kB 00:00 2024-03-25T16:21:28.438 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.0 MB/s | 863 kB 00:00 2024-03-25T16:21:28.805 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 57 kB/s | 19 kB 00:00 2024-03-25T16:21:29.147 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 671 kB/s | 548 kB 00:00 2024-03-25T16:21:29.273 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T16:21:29.373 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-25T16:21:29.415 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-25T16:21:29.498 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-25T16:21:31.099 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 526 kB/s | 1.8 MB 00:03 2024-03-25T16:21:31.104 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:31.104 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.8 MB/s | 6.9 MB 00:03 2024-03-25T16:21:31.124 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:21:31.143 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:21:31.144 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:21:31.324 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:21:31.324 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:21:31.830 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:21:31.899 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:21:31.942 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:21:31.989 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:21:32.028 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:21:32.137 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:21:32.269 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:21:32.356 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:21:32.407 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:21:32.436 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:21:32.467 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:21:32.709 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:21:32.742 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:21:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:21:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:21:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:21:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:21:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:21:33.570 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:21:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:21:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:21:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:21:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:21:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:21:33.571 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:21:33.917 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:21:33.917 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:33.917 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:21:33.917 INFO:teuthology.orchestra.run.smithi191.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:33.918 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:34.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:21:34.915 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:02:50 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:35.065 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:35.066 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-25T16:21:35.066 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:21:35.067 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:21:35.068 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:21:35.068 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:35.068 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:21:35.068 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2024-03-25T16:21:35.068 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-03-25T16:21:35.068 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:35.068 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 37 M 2024-03-25T16:21:35.069 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 209 M 2024-03-25T16:21:35.069 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:21:35.337 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 304 kB/s | 32 kB 00:00 2024-03-25T16:21:35.388 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 646 kB/s | 17 kB 00:00 2024-03-25T16:21:35.421 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:21:35.580 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T16:21:35.722 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T16:21:35.789 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.4 MB/s | 3.0 MB 00:00 2024-03-25T16:21:35.814 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-25T16:21:36.122 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-25T16:21:36.708 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 334 kB/s | 195 kB 00:00 2024-03-25T16:21:36.742 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 868 kB/s | 803 kB 00:00 2024-03-25T16:21:36.867 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-25T16:21:40.425 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.4 MB/s | 7.4 MB 00:05 2024-03-25T16:21:40.427 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:40.428 INFO:teuthology.orchestra.run.smithi191.stdout:Total 6.9 MB/s | 37 MB 00:05 2024-03-25T16:21:40.472 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:21:40.497 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:21:40.497 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:21:40.863 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:21:40.863 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:21:41.567 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:21:41.631 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:21:41.662 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:21:41.754 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:21:41.808 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:21:41.984 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:21:42.015 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:21:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:21:42.730 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:21:42.851 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:21:44.086 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:21:45.100 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:21:45.139 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:21:47.856 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:21:47.857 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:21:48.249 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:21:48.250 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:21:48.250 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:21:48.250 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:48.250 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:48.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:21:49.219 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:05 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:49.370 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:49.371 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:21:49.371 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:21:49.371 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:21:49.371 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:21:49.371 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:49.372 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:21:49.373 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:21:49.373 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2024-03-25T16:21:49.373 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:49.373 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.0 M 2024-03-25T16:21:49.373 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 26 M 2024-03-25T16:21:49.373 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:21:49.555 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 218 kB/s | 24 kB 00:00 2024-03-25T16:21:49.605 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-03-25T16:21:49.655 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:21:49.681 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:21:49.727 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:21:49.731 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 869 kB/s | 43 kB 00:00 2024-03-25T16:21:50.023 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 102 kB/s | 32 kB 00:00 2024-03-25T16:21:50.049 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 47 kB/s | 15 kB 00:00 2024-03-25T16:21:50.074 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 83 kB/s | 52 kB 00:00 2024-03-25T16:21:50.266 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 836 kB/s | 203 kB 00:00 2024-03-25T16:21:50.367 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 896 kB/s | 90 kB 00:00 2024-03-25T16:21:50.408 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 177 kB/s | 59 kB 00:00 2024-03-25T16:21:50.475 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-03-25T16:21:50.479 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:50.479 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-03-25T16:21:50.480 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:21:50.493 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:21:50.493 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:21:50.582 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:21:50.582 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:21:51.103 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:21:51.280 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:21:51.334 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:21:51.390 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:21:51.427 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:21:51.468 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:21:51.563 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:21:51.671 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:21:51.736 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:21:52.453 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:21:52.500 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:21:52.535 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:21:52.582 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:21:52.604 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:21:53.452 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:21:53.453 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:21:53.806 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:21:53.807 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:21:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:21:53.808 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:53.808 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:54.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:21:54.639 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:10 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:54.790 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:54.790 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:21:54.791 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-03-25T16:21:54.792 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:54.792 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 6.8 M 2024-03-25T16:21:54.792 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 33 M 2024-03-25T16:21:54.792 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:21:55.385 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-03-25T16:21:55.527 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 418 kB/s | 248 kB 00:00 2024-03-25T16:21:55.910 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 141 kB/s | 138 kB 00:00 2024-03-25T16:21:55.952 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 136 kB/s | 58 kB 00:00 2024-03-25T16:21:56.053 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 821 kB/s | 548 kB 00:00 2024-03-25T16:21:56.345 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 118 kB/s | 51 kB 00:00 2024-03-25T16:21:56.348 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:56.348 INFO:teuthology.orchestra.run.smithi191.stdout:Total 4.4 MB/s | 6.8 MB 00:01 2024-03-25T16:21:56.361 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:21:56.372 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:21:56.372 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:21:56.461 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:21:56.462 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:21:56.776 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:21:56.943 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:21:56.996 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:21:57.510 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:21:57.590 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:21:57.652 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:21:57.678 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:21:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:21:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:21:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:21:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:21:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:21:58.460 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:21:58.765 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:21:58.765 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:58.765 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:21:58.765 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:21:58.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:21:58.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:21:58.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:21:58.765 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:21:58.766 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:21:58.766 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:58.766 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:21:59.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2024-03-25T16:21:59.585 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:15 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:21:59.735 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-03-25T16:21:59.736 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:21:59.737 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 964 k 2024-03-25T16:21:59.737 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.8 M 2024-03-25T16:21:59.737 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:21:59.879 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 555 kB/s | 79 kB 00:00 2024-03-25T16:22:01.287 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 571 kB/s | 885 kB 00:01 2024-03-25T16:22:01.289 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:01.289 INFO:teuthology.orchestra.run.smithi191.stdout:Total 621 kB/s | 964 kB 00:01 2024-03-25T16:22:01.290 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:22:01.313 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:22:01.313 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:22:01.391 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:22:01.391 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:22:01.581 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:22:01.713 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:22:01.782 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:22:02.525 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:22:02.525 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:22:02.854 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:22:02.854 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:02.854 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:22:02.854 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:22:02.854 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:02.854 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:03.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-volume 2024-03-25T16:22:03.717 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:19 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:03.866 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:03.867 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:03.867 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-03-25T16:22:03.867 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:03.867 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 332 k 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.5 M 2024-03-25T16:22:03.868 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:22:03.983 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 600 kB/s | 69 kB 00:00 2024-03-25T16:22:05.300 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 184 kB/s | 264 kB 00:01 2024-03-25T16:22:05.301 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:05.301 INFO:teuthology.orchestra.run.smithi191.stdout:Total 232 kB/s | 332 kB 00:01 2024-03-25T16:22:05.302 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:22:05.311 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:22:05.312 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:22:05.359 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:22:05.359 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:22:05.512 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:22:05.698 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:22:05.735 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:22:05.751 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:22:05.751 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:05.751 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:22:05.752 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:06.484 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:22:06.829 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:22:06.829 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:06.829 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:22:06.829 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:22:06.829 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:22:06.829 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:06.829 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:07.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2024-03-25T16:22:07.759 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:23 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:07.912 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:07.913 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:07.913 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:22:07.913 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:07.913 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:22:07.913 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:22:07.913 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:07.913 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:22:07.914 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:07.914 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:22:07.914 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:07.914 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 127 k 2024-03-25T16:22:07.914 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 456 k 2024-03-25T16:22:07.914 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:22:08.864 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 133 kB/s | 127 kB 00:00 2024-03-25T16:22:08.864 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:08.865 INFO:teuthology.orchestra.run.smithi191.stdout:Total 133 kB/s | 127 kB 00:00 2024-03-25T16:22:08.865 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:22:08.871 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:22:08.871 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:22:08.929 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:22:08.930 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:22:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:22:09.188 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:09.660 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:09.940 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:09.940 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:09.940 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:22:09.940 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:09.940 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:09.941 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:10.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2024-03-25T16:22:10.746 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:26 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:10.846 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:10.902 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:10.903 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:22:10.903 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:10.968 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2024-03-25T16:22:11.531 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:27 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:11.679 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:11.680 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 31 k 2024-03-25T16:22:11.681 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 140 k 2024-03-25T16:22:11.681 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:22:12.108 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 73 kB/s | 31 kB 00:00 2024-03-25T16:22:12.109 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:12.109 INFO:teuthology.orchestra.run.smithi191.stdout:Total 73 kB/s | 31 kB 00:00 2024-03-25T16:22:12.109 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:22:12.113 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:22:12.113 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:22:12.140 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:22:12.141 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:22:12.267 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:22:12.373 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:12.753 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:13.027 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:13.027 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:13.028 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:22:13.028 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:13.028 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:13.028 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:13.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2024-03-25T16:22:13.785 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:13.888 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:13.939 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:13.940 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:22:13.941 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:14.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2024-03-25T16:22:14.570 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:30 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:14.668 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:14.719 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:14.721 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:22:14.721 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:14.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2024-03-25T16:22:15.354 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:31 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:15.458 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:15.513 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:15.514 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:22:15.514 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:15.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2024-03-25T16:22:16.149 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:32 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:16.247 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:16.300 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:16.301 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:22:16.301 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:16.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2024-03-25T16:22:16.918 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:32 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:17.018 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:17.070 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:17.071 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:22:17.071 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:17.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2024-03-25T16:22:17.690 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:33 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:17.788 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:22:17.840 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:17.841 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T16:22:17.841 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:17.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2024-03-25T16:22:18.456 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:34 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:18.604 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:18.605 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-25T16:22:18.605 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:18.605 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:22:18.605 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:22:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:18.606 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:22:18.606 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:18.606 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:22:18.606 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:18.606 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 85 k 2024-03-25T16:22:18.606 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 231 k 2024-03-25T16:22:18.606 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:22:19.170 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 151 kB/s | 85 kB 00:00 2024-03-25T16:22:19.170 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:19.170 INFO:teuthology.orchestra.run.smithi191.stdout:Total 151 kB/s | 85 kB 00:00 2024-03-25T16:22:19.171 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:22:19.178 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:22:19.178 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:22:19.233 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:22:19.234 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:22:19.369 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:22:19.460 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:19.886 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:20.194 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:20.195 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:20.195 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:22:20.195 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:20.195 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:20.195 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:20.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-03-25T16:22:21.002 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:37 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:21.154 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:21.155 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.1 M 2024-03-25T16:22:21.156 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 13 M 2024-03-25T16:22:21.156 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:22:25.530 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 723 kB/s | 3.1 MB 00:04 2024-03-25T16:22:25.531 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:25.531 INFO:teuthology.orchestra.run.smithi191.stdout:Total 722 kB/s | 3.1 MB 00:04 2024-03-25T16:22:25.532 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:22:25.543 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:22:25.543 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:22:25.618 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:22:25.619 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:22:26.063 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:22:26.156 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:26.180 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:26.180 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:22:26.180 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:22:26.180 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:22:26.180 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:22:26.181 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:27.304 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:27.305 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:22:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:27.305 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:27.305 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:27.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2024-03-25T16:22:28.119 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:44 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:28.267 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:22:28.268 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:28.269 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 171 k 2024-03-25T16:22:28.269 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 494 k 2024-03-25T16:22:28.269 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:22:29.033 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 224 kB/s | 171 kB 00:00 2024-03-25T16:22:29.034 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:29.034 INFO:teuthology.orchestra.run.smithi191.stdout:Total 224 kB/s | 171 kB 00:00 2024-03-25T16:22:29.034 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:22:29.042 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:22:29.042 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:22:29.099 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:22:29.099 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:22:29.249 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:22:29.357 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:29.791 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:30.091 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:22:30.091 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:30.091 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:22:30.091 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:30.091 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:30.092 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:22:30.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:22:30.880 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:03:46 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:22:31.032 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:22:31.033 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:31.033 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:22:31.033 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:31.033 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 91 M 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 498 M 2024-03-25T16:22:31.034 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:22:57.107 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 882 kB/s | 22 MB 00:26 2024-03-25T16:23:27.996 INFO:teuthology.orchestra.run.smithi162.stdout:[MIRROR] grpc-data-1.46.7-10.el9.noarch.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/g/grpc-data-1.46.7-10.el9.noarch.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-25T16:24:32.427 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): grpc-data-1.46.7-10.el9.noarch.rpm 102 B/s | 19 kB 03:15 2024-03-25T16:24:32.428 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:32.428 INFO:teuthology.orchestra.run.smithi162.stdout:Total 36 kB/s | 6.9 MB 03:16 2024-03-25T16:24:32.446 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:24:32.467 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:24:32.467 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:24:32.651 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:24:32.651 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:24:33.172 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:24:33.244 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:24:33.279 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:24:33.326 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:24:33.356 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:24:33.445 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:24:33.574 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:24:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:24:33.711 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:24:33.748 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:24:33.780 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:24:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:24:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:24:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:24:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:24:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:24:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:24:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:24:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:24:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:24:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:24:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:24:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:24:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:24:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:24:35.143 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:24:35.144 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:24:35.145 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:24:35.145 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:24:35.145 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:24:35.145 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:24:35.145 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:24:35.145 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:24:35.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:24:35.990 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:24:36.149 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:24:36.150 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-25T16:24:36.150 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:24:36.150 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-25T16:24:36.150 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:24:36.150 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:24:36.150 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-25T16:24:36.151 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-03-25T16:24:36.152 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:24:36.152 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 37 M 2024-03-25T16:24:36.152 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 209 M 2024-03-25T16:24:36.152 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:24:36.552 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2024-03-25T16:24:36.594 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 417 kB/s | 17 kB 00:00 2024-03-25T16:24:36.661 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T16:24:36.686 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 386 kB/s | 34 kB 00:00 2024-03-25T16:24:37.303 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.4 MB/s | 4.7 MB 00:00 2024-03-25T16:24:37.436 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.4 MB/s | 4.8 MB 00:00 2024-03-25T16:24:37.462 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 217 kB 00:00 2024-03-25T16:24:38.087 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-03-25T16:24:38.312 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 945 kB/s | 803 kB 00:00 2024-03-25T16:24:38.496 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T16:24:38.588 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 390 kB/s | 195 kB 00:00 2024-03-25T16:24:58.396 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 473 kB/s | 68 MB 02:27 2024-03-25T16:24:58.397 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:58.398 INFO:teuthology.orchestra.run.smithi191.stdout:Total 629 kB/s | 91 MB 02:27 2024-03-25T16:24:58.398 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:24:58.402 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:24:58.402 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:24:58.990 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:24:58.991 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:25:00.617 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 312 kB/s | 7.4 MB 00:24 2024-03-25T16:25:00.620 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:00.620 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.5 MB/s | 37 MB 00:24 2024-03-25T16:25:00.662 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:00.687 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:00.687 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:01.049 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:01.050 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:01.765 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:01.831 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:25:01.855 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:25:01.949 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:25:01.991 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:25:02.101 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:25:02.170 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:25:02.232 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:25:02.315 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:25:02.959 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:25:03.070 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:25:04.317 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:25:05.554 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:25:05.591 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:25:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:25:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:25:08.579 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:25:08.580 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:25:08.581 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:25:08.581 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:25:08.581 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:08.581 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:08.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:25:09.232 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:25:09.305 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:09.522 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:20 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:09.672 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:09.673 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:25:09.673 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:25:09.673 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:25:09.673 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:09.673 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:25:09.673 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:25:09.673 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:09.674 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:09.675 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:25:09.675 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-03-25T16:25:09.675 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:09.675 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.0 M 2024-03-25T16:25:09.675 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2024-03-25T16:25:09.675 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:25:09.985 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 165 kB/s | 18 kB 00:00 2024-03-25T16:25:10.027 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 161 kB/s | 24 kB 00:00 2024-03-25T16:25:10.169 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2024-03-25T16:25:10.260 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 566 kB/s | 132 kB 00:00 2024-03-25T16:25:10.336 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 113 kB/s | 52 kB 00:00 2024-03-25T16:25:10.378 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.0 MB/s | 211 kB 00:00 2024-03-25T16:25:10.403 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 300 kB/s | 43 kB 00:00 2024-03-25T16:25:10.820 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 66 kB/s | 32 kB 00:00 2024-03-25T16:25:10.845 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 32 kB/s | 15 kB 00:00 2024-03-25T16:25:11.029 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-03-25T16:25:11.137 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 276 kB/s | 203 kB 00:00 2024-03-25T16:25:11.246 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 414 kB/s | 90 kB 00:00 2024-03-25T16:25:12.146 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 788 kB/s | 1.0 MB 00:01 2024-03-25T16:25:12.152 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:12.152 INFO:teuthology.orchestra.run.smithi162.stdout:Total 847 kB/s | 2.0 MB 00:02 2024-03-25T16:25:12.153 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:12.169 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:12.169 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:12.265 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:12.265 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:12.778 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:12.928 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:25:12.928 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:25:12.952 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:25:13.023 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:25:13.078 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:25:13.116 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:25:13.157 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:25:13.194 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:25:13.194 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:25:13.194 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:25:13.195 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:13.195 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:13.195 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:25:13.195 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:25:13.251 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:25:13.377 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:25:13.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:25:13.425 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:25:13.966 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:06:29 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:25:14.115 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:25:14.116 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:25:14.116 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:25:14.116 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:25:14.116 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:25:14.116 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:25:14.116 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:25:14.117 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:25:14.117 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:25:14.117 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:25:14.117 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:25:14.117 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 416 M 2024-03-25T16:25:14.117 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.7 G 2024-03-25T16:25:14.117 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:25:14.147 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:25:14.198 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:25:14.231 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:25:14.280 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:25:14.301 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:25:15.117 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:25:15.117 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:25:15.118 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:25:15.478 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:25:15.479 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:25:15.479 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:25:15.479 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:25:15.479 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:25:15.479 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:25:15.479 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:25:15.479 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:15.479 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:15.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:25:16.316 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:27 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:16.467 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:16.468 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:25:16.468 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:25:16.468 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:25:16.468 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-03-25T16:25:16.469 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:16.470 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 6.8 M 2024-03-25T16:25:16.470 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 33 M 2024-03-25T16:25:16.470 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:25:16.833 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-25T16:25:16.958 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-25T16:25:17.450 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 124 kB/s | 58 kB 00:00 2024-03-25T16:25:17.534 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 157 kB/s | 138 kB 00:00 2024-03-25T16:25:17.593 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 359 kB/s | 51 kB 00:00 2024-03-25T16:25:17.626 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 693 kB/s | 548 kB 00:00 2024-03-25T16:25:17.629 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:17.629 INFO:teuthology.orchestra.run.smithi162.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2024-03-25T16:25:17.642 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:17.652 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:17.652 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:17.743 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:17.743 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:18.057 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:18.224 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:25:18.277 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:25:18.804 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:25:18.889 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:25:18.958 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:25:18.982 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:25:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:25:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:25:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:25:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:25:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:25:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:25:20.100 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:20.101 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:20.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-03-25T16:25:20.908 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:31 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:21.059 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:21.060 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-03-25T16:25:21.061 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 964 k 2024-03-25T16:25:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.8 M 2024-03-25T16:25:21.061 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:25:21.195 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 592 kB/s | 79 kB 00:00 2024-03-25T16:25:24.336 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 270 kB/s | 885 kB 00:03 2024-03-25T16:25:24.338 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:24.338 INFO:teuthology.orchestra.run.smithi162.stdout:Total 294 kB/s | 964 kB 00:03 2024-03-25T16:25:24.339 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:24.362 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:24.363 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:24.439 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:24.439 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:24.626 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:24.763 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:25:24.828 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:25:25.476 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:25:25.476 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:25.797 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:26.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-volume 2024-03-25T16:25:26.667 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:37 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:26.819 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:26.819 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:26.819 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-25T16:25:26.819 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-03-25T16:25:26.820 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:26.821 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 332 k 2024-03-25T16:25:26.821 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.5 M 2024-03-25T16:25:26.821 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:25:26.934 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 611 kB/s | 69 kB 00:00 2024-03-25T16:25:28.100 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 206 kB/s | 264 kB 00:01 2024-03-25T16:25:28.102 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:28.103 INFO:teuthology.orchestra.run.smithi162.stdout:Total 259 kB/s | 332 kB 00:01 2024-03-25T16:25:28.103 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:28.112 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:28.112 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:28.162 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:28.162 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:28.517 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:25:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:25:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:25:28.569 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:28.569 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:25:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:29.286 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:25:29.626 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:25:29.626 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:29.626 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:29.627 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:29.627 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:25:29.627 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:29.627 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:29.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-03-25T16:25:30.520 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:41 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:30.670 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:30.671 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:30.671 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:25:30.671 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:30.671 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:30.671 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:25:30.671 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:30.671 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:30.671 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:30.672 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:25:30.672 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:30.672 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 127 k 2024-03-25T16:25:30.672 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 456 k 2024-03-25T16:25:30.672 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:25:31.491 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 154 kB/s | 127 kB 00:00 2024-03-25T16:25:31.491 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:31.492 INFO:teuthology.orchestra.run.smithi162.stdout:Total 154 kB/s | 127 kB 00:00 2024-03-25T16:25:31.492 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:31.498 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:31.498 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:31.555 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:31.555 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:31.699 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:31.815 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:32.257 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:32.548 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:32.548 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:32.548 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:32.548 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:32.548 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:32.548 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:32.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-03-25T16:25:33.356 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:44 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:33.461 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:33.518 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:33.519 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:25:33.519 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:33.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-03-25T16:25:34.157 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:45 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:34.306 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 31 k 2024-03-25T16:25:34.307 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 140 k 2024-03-25T16:25:34.308 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:25:34.764 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 68 kB/s | 31 kB 00:00 2024-03-25T16:25:34.765 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:34.765 INFO:teuthology.orchestra.run.smithi162.stdout:Total 68 kB/s | 31 kB 00:00 2024-03-25T16:25:34.765 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:34.768 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:34.768 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:34.803 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:34.803 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:34.937 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:35.063 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:35.455 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:35.713 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:35.713 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:35.713 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:35.714 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:35.714 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:35.714 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:35.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-03-25T16:25:36.472 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:47 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:36.573 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:36.627 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:36.628 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:25:36.628 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:36.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-03-25T16:25:37.286 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:48 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:37.387 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:37.441 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:37.442 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:25:37.442 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:37.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-03-25T16:25:38.064 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:49 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:38.159 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:38.211 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:38.211 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:25:38.212 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:38.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-03-25T16:25:38.829 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:49 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:38.925 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:38.976 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:38.977 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:25:38.978 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:39.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-03-25T16:25:39.599 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:50 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:39.697 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:39.748 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:39.749 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:25:39.749 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:39.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-03-25T16:25:40.368 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:51 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:40.465 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:40.516 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:40.517 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T16:25:40.517 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:40.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-03-25T16:25:41.139 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:52 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:41.288 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:41.289 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:41.289 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-03-25T16:25:41.289 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:41.289 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:41.289 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 85 k 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 231 k 2024-03-25T16:25:41.290 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:25:41.892 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 142 kB/s | 85 kB 00:00 2024-03-25T16:25:41.892 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:41.893 INFO:teuthology.orchestra.run.smithi162.stdout:Total 141 kB/s | 85 kB 00:00 2024-03-25T16:25:41.893 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:41.900 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:41.900 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:41.958 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:41.959 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:42.094 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:42.191 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:42.580 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:42.885 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:42.886 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:42.886 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:42.886 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:42.886 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:42.886 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:43.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-03-25T16:25:43.681 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:06:54 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:43.835 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:43.836 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:25:43.837 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:43.837 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.1 M 2024-03-25T16:25:43.837 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 13 M 2024-03-25T16:25:43.837 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:25:51.522 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 412 kB/s | 3.1 MB 00:07 2024-03-25T16:25:51.522 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:51.522 INFO:teuthology.orchestra.run.smithi162.stdout:Total 411 kB/s | 3.1 MB 00:07 2024-03-25T16:25:51.523 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:25:51.532 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:25:51.532 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:25:51.608 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:25:51.608 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:25:52.029 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:25:52.095 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:52.119 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:52.119 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:52.119 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:25:52.120 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:25:52.120 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:25:52.120 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:53.242 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:25:53.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-03-25T16:25:54.045 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:07:05 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:25:54.196 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:25:54.197 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 171 k 2024-03-25T16:25:54.198 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 494 k 2024-03-25T16:25:54.198 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:26:01.972 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 22 kB/s | 171 kB 00:07 2024-03-25T16:26:01.973 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:01.973 INFO:teuthology.orchestra.run.smithi162.stdout:Total 22 kB/s | 171 kB 00:07 2024-03-25T16:26:01.974 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:26:01.983 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:26:01.983 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:26:02.044 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:26:02.045 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:26:02.199 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:26:02.304 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:02.751 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:03.028 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:03.028 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:26:03.028 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:26:03.028 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:03.028 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:26:03.029 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:26:03.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:26:03.875 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:07:14 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:26:04.030 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:26:04.030 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:26:04.030 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:26:04.030 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:26:04.030 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:26:04.030 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:26:04.030 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-25T16:26:04.030 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:26:04.031 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:26:04.031 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:26:04.031 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:26:04.031 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-03-25T16:26:04.031 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:26:04.031 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 91 M 2024-03-25T16:26:04.031 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 498 M 2024-03-25T16:26:04.031 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:26:39.253 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 653 kB/s | 22 MB 00:35 2024-03-25T16:28:56.892 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 404 kB/s | 68 MB 02:52 2024-03-25T16:28:56.893 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:56.893 INFO:teuthology.orchestra.run.smithi162.stdout:Total 537 kB/s | 91 MB 02:52 2024-03-25T16:28:56.893 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:28:56.897 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:28:56.897 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:28:57.486 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:28:57.486 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:29:00.640 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:29:07.789 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:29:07.858 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:29:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:29:11.255 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:29:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:29:11.515 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:29:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:29:11.515 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:29:11.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:29:12.287 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:10:23 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:29:12.435 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 416 M 2024-03-25T16:29:12.436 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.7 G 2024-03-25T16:29:12.437 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:43:21.941 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 502 kB/s | 416 MB 14:09 2024-03-25T16:43:21.950 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:43:21.951 INFO:teuthology.orchestra.run.smithi162.stdout:Total 502 kB/s | 416 MB 14:09 2024-03-25T16:43:21.951 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:43:21.951 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:43:21.951 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:43:24.407 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:43:24.408 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:44:06.509 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:44:06.624 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:44:11.276 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:44:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:44:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:44:11.527 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:44:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:44:11.528 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:44:11.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:44:12.345 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:25:23 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:44:12.494 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:44:12.495 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:44:12.495 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:44:12.495 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:44:12.495 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:44:12.495 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:44:12.495 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:44:12.496 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:44:12.496 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-25T16:44:12.496 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:44:12.496 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:44:12.496 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.4 M 2024-03-25T16:44:12.496 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 6.4 M 2024-03-25T16:44:12.496 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:44:20.818 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 176 kB/s | 1.4 MB 00:08 2024-03-25T16:44:20.819 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:44:20.819 INFO:teuthology.orchestra.run.smithi162.stdout:Total 176 kB/s | 1.4 MB 00:08 2024-03-25T16:44:20.819 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:44:20.823 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:44:20.823 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:44:20.867 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:44:20.868 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:44:21.127 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:44:21.233 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:44:21.626 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:44:21.880 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:44:21.880 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:44:21.880 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:44:21.880 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:44:21.880 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:44:21.880 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:44:22.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:44:22.641 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:25:33 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:44:22.789 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:44:22.790 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:44:22.790 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 176 M 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 753 M 2024-03-25T16:44:22.791 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T16:48:55.314 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 300 kB/s | 416 MB 23:41 2024-03-25T16:48:55.314 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:48:55.314 INFO:teuthology.orchestra.run.smithi191.stdout:Total 300 kB/s | 416 MB 23:41 2024-03-25T16:48:55.314 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:48:55.317 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:48:55.317 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:48:57.778 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:48:57.778 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:49:39.908 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:49:40.498 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:49:49.870 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:49:50.122 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:49:50.122 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:49:50.123 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:49:50.123 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:49:50.123 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:49:50.123 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:49:50.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:49:50.966 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:31:06 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:49:51.129 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================== 2024-03-25T16:49:51.130 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:49:51.131 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:49:51.131 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 1.4 M 2024-03-25T16:49:51.131 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 6.4 M 2024-03-25T16:49:51.131 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:49:55.141 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 364 kB/s | 1.4 MB 00:04 2024-03-25T16:49:55.143 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:49:55.143 INFO:teuthology.orchestra.run.smithi191.stdout:Total 364 kB/s | 1.4 MB 00:04 2024-03-25T16:49:55.143 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:49:55.147 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:49:55.147 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:49:55.193 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:49:55.193 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:49:55.453 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:49:55.548 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:49:55.989 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:49:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:49:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:49:56.287 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:49:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:49:56.287 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:49:56.287 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:49:56.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:49:57.041 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:31:13 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:49:57.190 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:49:57.190 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:49:57.190 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:49:57.190 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:49:57.190 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 176 M 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 753 M 2024-03-25T16:49:57.191 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:56:34.881 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 452 kB/s | 176 MB 06:37 2024-03-25T16:56:34.882 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:34.882 INFO:teuthology.orchestra.run.smithi191.stdout:Total 452 kB/s | 176 MB 06:37 2024-03-25T16:56:34.882 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T16:56:34.885 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T16:56:34.885 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T16:56:35.944 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T16:56:35.944 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T16:56:47.887 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 241 kB/s | 176 MB 12:25 2024-03-25T16:56:47.888 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:47.888 INFO:teuthology.orchestra.run.smithi162.stdout:Total 241 kB/s | 176 MB 12:25 2024-03-25T16:56:47.888 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T16:56:47.892 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T16:56:47.892 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T16:56:48.952 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T16:56:48.952 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T16:56:53.850 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T16:56:53.942 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:56:58.951 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:56:59.212 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:56:59.212 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:56:59.212 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T16:56:59.212 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:59.212 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:56:59.212 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T16:56:59.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:56:59.989 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:38:15 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T16:57:00.140 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 672 M 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 3.1 G 2024-03-25T16:57:00.141 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T16:57:07.220 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T16:57:07.324 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:11.962 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:12.228 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:57:12.228 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:57:12.228 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T16:57:12.228 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:12.228 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:57:12.228 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T16:57:12.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:57:13.003 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:38:24 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T16:57:13.151 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T16:57:13.152 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T16:57:13.153 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 672 M 2024-03-25T16:57:13.153 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 3.1 G 2024-03-25T16:57:13.153 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T17:34:02.555 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 312 kB/s | 672 MB 36:49 2024-03-25T17:34:02.577 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:34:02.577 INFO:teuthology.orchestra.run.smithi162.stdout:Total 312 kB/s | 672 MB 36:49 2024-03-25T17:34:02.577 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T17:34:02.577 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T17:34:02.577 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T17:34:06.513 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T17:34:06.513 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T17:35:17.313 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T17:35:17.477 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:26.699 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:35:26.968 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T17:35:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:35:26.968 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:35:26.968 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:35:27.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:35:27.824 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:16:38 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:35:27.922 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:35:27.974 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:35:27.975 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-25T17:35:27.976 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:35:28.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:35:28.619 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:16:39 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:35:28.767 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:35:28.768 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:35:28.768 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T17:35:28.768 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:35:28.768 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T17:35:28.768 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:35:28.769 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:35:28.769 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T17:35:28.769 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:35:28.769 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T17:35:28.769 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:35:28.769 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 20 M 2024-03-25T17:35:28.769 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 103 M 2024-03-25T17:35:28.769 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T17:36:22.261 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 377 kB/s | 20 MB 00:53 2024-03-25T17:36:22.261 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:36:22.262 INFO:teuthology.orchestra.run.smithi162.stdout:Total 377 kB/s | 20 MB 00:53 2024-03-25T17:36:22.262 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T17:36:22.266 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T17:36:22.266 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T17:36:22.415 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T17:36:22.416 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T17:36:24.695 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T17:36:24.867 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:25.869 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:26.128 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:36:26.128 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:36:26.128 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T17:36:26.128 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:36:26.128 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:36:26.128 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:36:26.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:36:26.889 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:17:37 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:36:27.037 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:36:27.037 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:36:27.037 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T17:36:27.037 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:36:27.037 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T17:36:27.037 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:36:27.038 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:36:27.038 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T17:36:27.038 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:36:27.038 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T17:36:27.038 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:36:27.038 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 48 M 2024-03-25T17:36:27.038 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 173 M 2024-03-25T17:36:27.038 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T17:38:13.938 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 278 kB/s | 672 MB 41:13 2024-03-25T17:38:13.939 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:13.939 INFO:teuthology.orchestra.run.smithi191.stdout:Total 278 kB/s | 672 MB 41:13 2024-03-25T17:38:13.939 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T17:38:13.944 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T17:38:13.944 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T17:38:17.898 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T17:38:17.898 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T17:38:46.377 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 351 kB/s | 48 MB 02:19 2024-03-25T17:38:46.378 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:46.378 INFO:teuthology.orchestra.run.smithi162.stdout:Total 351 kB/s | 48 MB 02:19 2024-03-25T17:38:46.378 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T17:38:46.381 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T17:38:46.382 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T17:38:46.696 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T17:38:46.696 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T17:38:51.440 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T17:38:51.542 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:52.883 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:38:53.145 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T17:38:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:53.145 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:38:53.145 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:38:53.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:38:53.907 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:20:04 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:38:54.055 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:38:54.056 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T17:38:54.057 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:38:54.057 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 99 M 2024-03-25T17:38:54.057 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 449 M 2024-03-25T17:38:54.057 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T17:39:28.618 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T17:39:29.156 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:39:37.874 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:39:38.148 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:39:38.148 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:39:38.148 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T17:39:38.148 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:39:38.148 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:39:38.148 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T17:39:38.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:39:39.040 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:20:55 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T17:39:39.135 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:39:39.187 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T17:39:39.188 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-03-25T17:39:39.188 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T17:39:39.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:39:39.818 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:20:55 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T17:39:39.967 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 20 M 2024-03-25T17:39:39.968 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 103 M 2024-03-25T17:39:39.969 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T17:40:34.843 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 368 kB/s | 20 MB 00:54 2024-03-25T17:40:34.844 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:40:34.844 INFO:teuthology.orchestra.run.smithi191.stdout:Total 368 kB/s | 20 MB 00:54 2024-03-25T17:40:34.844 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T17:40:34.848 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T17:40:34.848 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T17:40:34.998 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T17:40:34.999 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T17:40:37.224 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T17:40:37.333 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:38.356 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T17:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:40:38.622 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T17:40:38.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:40:39.346 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:21:55 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T17:40:39.495 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T17:40:39.496 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:40:39.497 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 48 M 2024-03-25T17:40:39.497 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 173 M 2024-03-25T17:40:39.497 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T17:43:07.750 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 330 kB/s | 48 MB 02:28 2024-03-25T17:43:07.751 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:07.751 INFO:teuthology.orchestra.run.smithi191.stdout:Total 330 kB/s | 48 MB 02:28 2024-03-25T17:43:07.751 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T17:43:07.754 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T17:43:07.754 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T17:43:08.066 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T17:43:08.067 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T17:43:12.815 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T17:43:12.906 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:14.373 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:14.639 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:43:14.640 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T17:43:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:14.640 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:43:14.640 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T17:43:14.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:43:15.369 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:24:31 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T17:43:15.518 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T17:43:15.519 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:43:15.519 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 99 M 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 449 M 2024-03-25T17:43:15.520 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T17:43:33.606 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 361 kB/s | 99 MB 04:39 2024-03-25T17:43:33.607 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:33.607 INFO:teuthology.orchestra.run.smithi162.stdout:Total 361 kB/s | 99 MB 04:39 2024-03-25T17:43:33.607 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T17:43:33.611 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T17:43:33.611 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T17:43:34.220 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T17:43:34.220 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T17:43:44.522 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T17:43:44.629 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:47.763 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:47.763 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:43:47.763 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T17:43:47.763 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:47.764 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:43:47.764 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:43:47.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:43:48.527 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:24:59 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:43:48.676 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T17:43:48.677 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:43:48.678 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 287 M 2024-03-25T17:43:48.678 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.2 G 2024-03-25T17:43:48.678 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T17:48:05.051 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 348 kB/s | 99 MB 04:49 2024-03-25T17:48:05.051 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:05.052 INFO:teuthology.orchestra.run.smithi191.stdout:Total 348 kB/s | 99 MB 04:49 2024-03-25T17:48:05.052 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T17:48:05.054 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T17:48:05.055 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T17:48:05.661 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T17:48:05.662 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T17:48:15.885 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T17:48:15.980 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:19.442 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:48:19.442 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:48:19.442 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T17:48:19.442 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:48:19.442 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:48:19.442 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T17:48:19.628 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:48:20.191 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:29:36 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T17:48:20.338 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T17:48:20.339 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:48:20.339 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T17:48:20.339 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:48:20.339 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T17:48:20.339 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:48:20.339 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:48:20.339 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T17:48:20.339 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T17:48:20.340 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T17:48:20.340 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T17:48:20.340 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 287 M 2024-03-25T17:48:20.340 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.2 G 2024-03-25T17:48:20.340 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T17:56:16.340 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 393 kB/s | 287 MB 12:27 2024-03-25T17:56:16.340 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:16.340 INFO:teuthology.orchestra.run.smithi162.stdout:Total 393 kB/s | 287 MB 12:27 2024-03-25T17:56:16.341 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T17:56:16.343 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T17:56:16.344 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T17:56:18.056 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T17:56:18.056 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T17:56:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T17:56:47.816 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:55.440 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:55.729 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:56:55.729 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:56:55.729 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T17:56:55.729 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:56:55.729 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:56:55.729 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:56:55.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:56:56.509 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:38:07 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:56:56.658 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 5.9 M 2024-03-25T17:56:56.659 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 28 M 2024-03-25T17:56:56.660 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T17:57:11.242 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 417 kB/s | 5.9 MB 00:14 2024-03-25T17:57:11.243 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:11.243 INFO:teuthology.orchestra.run.smithi162.stdout:Total 417 kB/s | 5.9 MB 00:14 2024-03-25T17:57:11.244 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T17:57:11.247 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T17:57:11.247 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T17:57:11.319 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T17:57:11.319 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T17:57:12.050 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T17:57:12.157 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:12.701 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:12.963 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:12.963 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:12.963 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T17:57:12.963 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:12.963 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:12.963 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:57:13.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:57:13.709 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:38:24 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:57:13.857 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:57:13.858 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:13.858 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T17:57:13.858 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:13.858 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T17:57:13.858 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:57:13.859 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:13.859 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T17:57:13.859 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:13.859 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T17:57:13.859 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:13.859 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 109 k 2024-03-25T17:57:13.859 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 440 k 2024-03-25T17:57:13.859 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T17:57:14.778 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 118 kB/s | 109 kB 00:00 2024-03-25T17:57:14.779 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:14.779 INFO:teuthology.orchestra.run.smithi162.stdout:Total 118 kB/s | 109 kB 00:00 2024-03-25T17:57:14.779 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T17:57:14.782 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T17:57:14.783 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T17:57:14.817 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T17:57:14.818 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T17:57:14.939 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T17:57:15.043 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:15.409 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:15.671 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:15.671 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:15.671 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T17:57:15.671 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:15.671 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:15.671 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:57:15.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:57:16.404 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:38:27 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:57:16.553 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:57:16.554 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:16.554 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T17:57:16.554 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:16.554 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T17:57:16.554 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:57:16.555 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:16.555 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T17:57:16.555 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:16.555 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T17:57:16.555 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:16.555 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 5.4 M 2024-03-25T17:57:16.555 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 25 M 2024-03-25T17:57:16.555 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T17:57:35.011 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 297 kB/s | 5.4 MB 00:18 2024-03-25T17:57:35.012 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:35.012 INFO:teuthology.orchestra.run.smithi162.stdout:Total 297 kB/s | 5.4 MB 00:18 2024-03-25T17:57:35.012 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T17:57:35.016 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T17:57:35.016 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T17:57:35.084 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T17:57:35.084 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T17:57:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T17:57:35.869 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:57:36.399 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:57:36.667 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:57:36.667 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:36.667 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T17:57:36.667 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:36.667 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:36.667 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T17:57:36.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2-debuginfo 2024-03-25T17:57:37.446 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:38:48 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T17:57:37.595 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T17:57:37.596 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T17:57:37.596 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:37.596 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T17:57:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:57:37.596 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:37.596 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T17:57:37.596 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T17:57:37.597 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T17:57:37.597 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T17:57:37.597 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 44 M 2024-03-25T17:57:37.597 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 253 M 2024-03-25T17:57:37.597 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:00:02.103 INFO:teuthology.orchestra.run.smithi162.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 314 kB/s | 44 MB 02:24 2024-03-25T18:00:02.104 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:02.104 INFO:teuthology.orchestra.run.smithi162.stdout:Total 314 kB/s | 44 MB 02:24 2024-03-25T18:00:02.104 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:00:02.107 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:00:02.107 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:00:02.406 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:00:02.406 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:00:07.436 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:00:07.534 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:09.373 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:09.647 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:00:09.648 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:00:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:09.648 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:00:09.648 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:00:09.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:00:10.417 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:41:21 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T18:00:10.567 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T18:00:10.568 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:00:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T18:00:10.568 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:00:10.568 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T18:00:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:00:10.568 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:00:10.568 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T18:00:10.569 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:00:10.569 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T18:00:10.569 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:00:10.569 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 52 M 2024-03-25T18:00:10.569 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 308 M 2024-03-25T18:00:10.569 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:02:49.216 INFO:teuthology.orchestra.run.smithi162.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 335 kB/s | 52 MB 02:38 2024-03-25T18:02:49.216 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:49.216 INFO:teuthology.orchestra.run.smithi162.stdout:Total 335 kB/s | 52 MB 02:38 2024-03-25T18:02:49.217 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:02:49.220 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:02:49.220 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:02:49.563 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:02:49.563 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:02:55.523 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:02:55.748 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:02:57.911 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:02:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:02:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:02:58.179 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:02:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:58.179 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:02:58.179 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:02:58.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:02:58.943 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:44:09 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T18:02:59.091 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:02:59.092 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T18:02:59.093 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:02:59.093 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 615 k 2024-03-25T18:02:59.093 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.1 M 2024-03-25T18:02:59.093 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:03:00.859 INFO:teuthology.orchestra.run.smithi162.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 348 kB/s | 615 kB 00:01 2024-03-25T18:03:00.859 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:00.860 INFO:teuthology.orchestra.run.smithi162.stdout:Total 348 kB/s | 615 kB 00:01 2024-03-25T18:03:00.860 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:03:00.864 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:03:00.864 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:03:00.905 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:03:00.905 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:03:01.070 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:03:01.174 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:01.561 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:01.819 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:01.819 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:01.819 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:03:01.819 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:01.819 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:01.819 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:03:02.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:03:02.601 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:44:13 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T18:03:02.751 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T18:03:02.752 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:02.752 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T18:03:02.752 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:02.752 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 771 k 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.7 M 2024-03-25T18:03:02.753 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:03:06.711 INFO:teuthology.orchestra.run.smithi162.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 195 kB/s | 771 kB 00:03 2024-03-25T18:03:06.712 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:06.712 INFO:teuthology.orchestra.run.smithi162.stdout:Total 195 kB/s | 771 kB 00:03 2024-03-25T18:03:06.713 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:03:06.717 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:03:06.717 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:03:06.759 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:03:06.760 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:03:06.969 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:03:07.077 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:07.451 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:07.710 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:07.710 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:07.710 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:03:07.710 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:07.710 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:07.710 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:03:07.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:03:08.475 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:44:19 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T18:03:08.626 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T18:03:08.626 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.1 M 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 4.3 M 2024-03-25T18:03:08.627 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:03:10.406 INFO:teuthology.orchestra.run.smithi162.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 647 kB/s | 1.1 MB 00:01 2024-03-25T18:03:10.407 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:10.407 INFO:teuthology.orchestra.run.smithi162.stdout:Total 646 kB/s | 1.1 MB 00:01 2024-03-25T18:03:10.408 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:03:10.412 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:03:10.412 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:03:10.455 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:03:10.455 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:03:10.674 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:03:10.788 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:11.171 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:11.434 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:11.435 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:03:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:11.435 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:11.435 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:03:11.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:03:12.183 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 329 kB/s | 287 MB 14:51 2024-03-25T18:03:12.184 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:12.184 INFO:teuthology.orchestra.run.smithi191.stdout:Total 329 kB/s | 287 MB 14:51 2024-03-25T18:03:12.184 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:03:12.188 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:03:12.188 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:03:12.219 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:44:23 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T18:03:12.369 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T18:03:12.370 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 337 k 2024-03-25T18:03:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.0 M 2024-03-25T18:03:12.371 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:03:13.893 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:03:13.893 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:03:14.126 INFO:teuthology.orchestra.run.smithi162.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 192 kB/s | 337 kB 00:01 2024-03-25T18:03:14.127 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:14.127 INFO:teuthology.orchestra.run.smithi162.stdout:Total 192 kB/s | 337 kB 00:01 2024-03-25T18:03:14.127 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:03:14.131 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:03:14.131 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:03:14.171 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:03:14.171 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:03:14.309 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:03:14.418 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:14.783 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:15.041 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:15.041 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:15.041 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:03:15.041 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:15.041 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:15.041 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:03:15.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:03:15.826 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:44:26 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T18:03:15.976 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T18:03:15.976 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:15.976 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T18:03:15.976 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:15.976 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 509 k 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.5 M 2024-03-25T18:03:15.977 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:03:17.846 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 272 kB/s | 509 kB 00:01 2024-03-25T18:03:17.846 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:17.847 INFO:teuthology.orchestra.run.smithi162.stdout:Total 272 kB/s | 509 kB 00:01 2024-03-25T18:03:17.847 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:03:17.851 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:03:17.851 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:03:17.891 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:03:17.891 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:03:18.056 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:03:18.162 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:18.549 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:18.816 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:18.816 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:18.816 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:03:18.816 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:18.816 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:18.816 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:03:19.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:03:19.618 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:44:30 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T18:03:19.768 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T18:03:19.769 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T18:03:19.769 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:19.769 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T18:03:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:03:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:19.769 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T18:03:19.769 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:03:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T18:03:19.770 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:03:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 52 M 2024-03-25T18:03:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 298 M 2024-03-25T18:03:19.770 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:03:43.226 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:03:43.330 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:44.556 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:44.851 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:44.852 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:03:44.852 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:03:44.852 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:44.852 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:03:44.852 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:03:45.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:03:45.617 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:45:01 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:03:45.765 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:03:45.766 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:03:45.766 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:03:45.766 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:03:45.766 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:03:45.766 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:03:45.766 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:03:45.767 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:03:45.767 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:03:45.767 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:03:45.767 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:03:45.767 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 5.9 M 2024-03-25T18:03:45.767 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 28 M 2024-03-25T18:03:45.767 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:04:04.124 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 332 kB/s | 5.9 MB 00:18 2024-03-25T18:04:04.125 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:04.126 INFO:teuthology.orchestra.run.smithi191.stdout:Total 331 kB/s | 5.9 MB 00:18 2024-03-25T18:04:04.126 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:04:04.130 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:04:04.130 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:04:04.201 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:04:04.201 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:04:04.931 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:04:05.056 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:05.601 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:05.879 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:04:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:05.879 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:05.879 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:04:06.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:04:06.629 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:45:22 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:04:06.778 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:04:06.779 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:06.780 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 109 k 2024-03-25T18:04:06.780 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 440 k 2024-03-25T18:04:06.780 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:04:07.537 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 144 kB/s | 109 kB 00:00 2024-03-25T18:04:07.538 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:07.538 INFO:teuthology.orchestra.run.smithi191.stdout:Total 143 kB/s | 109 kB 00:00 2024-03-25T18:04:07.538 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:04:07.542 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:04:07.542 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:04:07.576 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:04:07.577 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:04:07.697 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:04:07.786 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:08.178 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:08.445 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:08.445 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:08.445 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:04:08.445 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:08.445 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:08.445 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:04:08.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:04:09.184 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:45:25 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:04:09.332 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:04:09.332 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:09.332 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:04:09.332 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:09.332 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:04:09.332 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:04:09.332 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:09.333 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:04:09.333 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:09.333 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:04:09.333 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:09.333 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 5.4 M 2024-03-25T18:04:09.333 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 25 M 2024-03-25T18:04:09.333 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:04:33.289 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 229 kB/s | 5.4 MB 00:23 2024-03-25T18:04:33.289 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:33.290 INFO:teuthology.orchestra.run.smithi191.stdout:Total 229 kB/s | 5.4 MB 00:23 2024-03-25T18:04:33.290 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:04:33.294 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:04:33.294 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:04:33.362 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:04:33.362 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:04:34.028 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:04:34.137 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:34.649 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:34.916 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:34.916 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:34.916 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:04:34.916 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:34.916 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:34.916 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:04:35.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2-debuginfo 2024-03-25T18:04:35.668 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:45:51 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:04:35.815 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:04:35.816 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:35.816 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:04:35.816 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 44 M 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 253 M 2024-03-25T18:04:35.817 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:05:11.410 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 473 kB/s | 52 MB 01:51 2024-03-25T18:05:11.411 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:11.411 INFO:teuthology.orchestra.run.smithi162.stdout:Total 473 kB/s | 52 MB 01:51 2024-03-25T18:05:11.411 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:05:11.415 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:05:11.415 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:05:11.748 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:05:11.749 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:05:17.624 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:05:17.983 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:20.069 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:20.338 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:05:20.339 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:05:20.339 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:05:20.339 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:20.339 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:05:20.339 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:05:20.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:05:21.107 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 1:46:32 ago on Mon 25 Mar 2024 04:18:49 PM UTC. 2024-03-25T18:05:21.256 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-25T18:05:21.257 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-25T18:05:21.258 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:05:21.258 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.3 M 2024-03-25T18:05:21.258 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 5.5 M 2024-03-25T18:05:21.258 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-25T18:05:25.628 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 293 kB/s | 1.3 MB 00:04 2024-03-25T18:05:25.628 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:25.629 INFO:teuthology.orchestra.run.smithi162.stdout:Total 293 kB/s | 1.3 MB 00:04 2024-03-25T18:05:25.629 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-25T18:05:25.633 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-25T18:05:25.633 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-25T18:05:25.679 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-25T18:05:25.679 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-25T18:05:25.938 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-25T18:05:26.043 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:26.467 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:26.729 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:26.729 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:05:26.730 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-25T18:05:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:26.730 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:05:26.730 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-25T18:05:26.941 DEBUG:teuthology.parallel:result is None 2024-03-25T18:06:47.323 INFO:teuthology.orchestra.run.smithi191.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 345 kB/s | 44 MB 02:11 2024-03-25T18:06:47.323 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:47.324 INFO:teuthology.orchestra.run.smithi191.stdout:Total 345 kB/s | 44 MB 02:11 2024-03-25T18:06:47.324 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:06:47.327 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:06:47.327 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:06:47.625 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:06:47.626 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:06:52.651 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:06:52.738 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:54.730 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:55.007 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:55.008 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:06:55.008 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:06:55.008 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:55.008 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:06:55.008 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:06:55.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:06:55.745 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:48:11 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:06:55.894 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:06:55.897 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:06:55.897 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:06:55.897 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:06:55.897 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 52 M 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 308 M 2024-03-25T18:06:55.898 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:10:05.718 INFO:teuthology.orchestra.run.smithi191.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 280 kB/s | 52 MB 03:09 2024-03-25T18:10:05.719 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:05.719 INFO:teuthology.orchestra.run.smithi191.stdout:Total 280 kB/s | 52 MB 03:09 2024-03-25T18:10:05.719 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:10:05.719 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:10:05.720 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:10:06.062 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:10:06.063 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:10:12.055 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:10:12.155 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:14.438 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:14.705 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:14.705 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:14.705 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:10:14.705 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:14.705 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:14.705 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:10:14.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:10:15.484 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:51:31 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:10:15.633 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:15.634 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 615 k 2024-03-25T18:10:15.635 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.1 M 2024-03-25T18:10:15.635 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:10:17.668 INFO:teuthology.orchestra.run.smithi191.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 302 kB/s | 615 kB 00:02 2024-03-25T18:10:17.669 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:17.670 INFO:teuthology.orchestra.run.smithi191.stdout:Total 302 kB/s | 615 kB 00:02 2024-03-25T18:10:17.670 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:10:17.674 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:10:17.674 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:10:17.713 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:10:17.713 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:10:17.880 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:10:17.983 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:18.391 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:18.655 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:18.655 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:18.655 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:10:18.655 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:18.655 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:18.655 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:10:18.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:10:19.423 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:51:35 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:10:19.574 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:10:19.575 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:19.576 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 771 k 2024-03-25T18:10:19.576 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.7 M 2024-03-25T18:10:19.576 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:10:21.964 INFO:teuthology.orchestra.run.smithi191.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 323 kB/s | 771 kB 00:02 2024-03-25T18:10:21.964 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:21.965 INFO:teuthology.orchestra.run.smithi191.stdout:Total 323 kB/s | 771 kB 00:02 2024-03-25T18:10:21.965 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:10:21.969 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:10:21.969 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:10:22.010 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:10:22.011 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:10:22.197 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:10:22.313 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:22.724 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:22.986 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:22.986 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:22.986 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:10:22.986 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:22.986 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:22.986 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:10:23.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:10:23.760 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:51:39 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:10:23.909 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:10:23.910 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:23.910 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:10:23.910 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:23.910 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:10:23.910 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:10:23.911 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:23.911 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:10:23.911 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:23.911 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:10:23.911 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:23.911 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 1.1 M 2024-03-25T18:10:23.911 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 4.3 M 2024-03-25T18:10:23.911 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:10:26.076 INFO:teuthology.orchestra.run.smithi191.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 531 kB/s | 1.1 MB 00:02 2024-03-25T18:10:26.077 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:26.077 INFO:teuthology.orchestra.run.smithi191.stdout:Total 531 kB/s | 1.1 MB 00:02 2024-03-25T18:10:26.078 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:10:26.081 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:10:26.082 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:10:26.124 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:10:26.124 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:10:26.341 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:10:26.452 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:26.858 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:27.138 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:27.139 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:27.139 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:10:27.139 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:27.139 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:27.139 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:10:27.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:10:27.918 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:51:43 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:10:28.069 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 337 k 2024-03-25T18:10:28.070 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.0 M 2024-03-25T18:10:28.071 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:10:29.822 INFO:teuthology.orchestra.run.smithi191.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 193 kB/s | 337 kB 00:01 2024-03-25T18:10:29.823 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:29.823 INFO:teuthology.orchestra.run.smithi191.stdout:Total 192 kB/s | 337 kB 00:01 2024-03-25T18:10:29.823 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:10:29.827 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:10:29.827 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:10:29.866 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:10:29.866 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:10:30.002 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:10:30.096 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:30.457 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:30.710 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:30.710 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:30.710 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:10:30.710 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:30.711 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:30.711 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:10:30.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:10:31.465 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:51:47 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:10:31.616 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:31.617 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:10:31.617 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:31.617 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 509 k 2024-03-25T18:10:31.617 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.5 M 2024-03-25T18:10:31.617 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:10:33.935 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 220 kB/s | 509 kB 00:02 2024-03-25T18:10:33.936 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:33.936 INFO:teuthology.orchestra.run.smithi191.stdout:Total 219 kB/s | 509 kB 00:02 2024-03-25T18:10:33.936 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:10:33.940 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:10:33.940 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:10:33.980 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:10:33.980 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:10:34.141 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:10:34.242 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:34.638 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:34.901 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:34.901 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:34.901 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:10:34.901 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:34.901 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:34.901 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:10:35.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:10:35.645 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:51:51 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:10:35.795 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:10:35.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:10:35.797 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 52 M 2024-03-25T18:10:35.797 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 298 M 2024-03-25T18:10:35.797 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:12:56.394 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 375 kB/s | 52 MB 02:20 2024-03-25T18:12:56.395 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:56.395 INFO:teuthology.orchestra.run.smithi191.stdout:Total 375 kB/s | 52 MB 02:20 2024-03-25T18:12:56.395 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:12:56.398 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:12:56.398 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:12:56.735 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:12:56.735 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:13:02.653 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:13:02.833 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:05.070 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:05.353 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:05.354 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:05.354 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:13:05.354 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:05.354 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:05.354 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:13:05.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:13:06.153 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 1:54:22 ago on Mon 25 Mar 2024 04:18:44 PM UTC. 2024-03-25T18:13:06.302 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-03-25T18:13:06.303 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-03-25T18:13:06.304 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:06.304 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 1.3 M 2024-03-25T18:13:06.304 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 5.5 M 2024-03-25T18:13:06.304 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-03-25T18:13:10.587 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 299 kB/s | 1.3 MB 00:04 2024-03-25T18:13:10.588 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:10.588 INFO:teuthology.orchestra.run.smithi191.stdout:Total 299 kB/s | 1.3 MB 00:04 2024-03-25T18:13:10.589 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-03-25T18:13:10.593 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-03-25T18:13:10.593 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-03-25T18:13:10.636 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-03-25T18:13:10.636 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-03-25T18:13:10.873 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-03-25T18:13:10.978 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:11.405 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:11.672 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:11.672 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:11.672 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-03-25T18:13:11.672 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:11.672 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:11.672 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-03-25T18:13:11.851 DEBUG:teuthology.parallel:result is None 2024-03-25T18:13:11.851 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:13:11.989 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:13:12.014 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:12.015 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:12.015 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:13:12.016 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:13:12.156 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:13:12.177 INFO:teuthology.orchestra.run.smithi191.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:12.177 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:12.177 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:13:12.179 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:13:12.179 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:12.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:13:12.211 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:12.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:13:12.245 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:13:12.245 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:12.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:13:12.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:13:12.352 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:12.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:13:12.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:13:12.446 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:13:12.447 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:12.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:13:12.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:13:12.544 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:12.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:13:12.570 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:13:12.637 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:13:12.637 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:12.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:13:12.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:13:12.732 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:12.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:13:12.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:13:12.825 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:13:12.902 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:13:12.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /var/log/ceph 2024-03-25T18:13:12.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2024-03-25T18:13:12.931 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:13:12.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:13:12.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:13:12.998 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:13:12.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:13:13.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:13:13.099 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:13:13.099 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:13:13.100 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621254', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi162.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmDuJIdzytL+sQK2u890CdR23URp81hjZiYk0h3GOYGZrh4ReOTPR/CTG9+Yf3U+ctPKclt7vHrQJeTXjsoR1A=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/wg/nRE65EJryN827OMAbsAXfPhKVsgN7uskCi6+GuQK9UkPx1JoN6arIrnJiWmNxcKP2nzDLkXe0PoG+Tv8Q='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:13:13.100 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:13:13.106 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:13:13.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:13:13.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:13:13.188 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:13.188 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:13:13.240 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:13:13.241 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-03-25T18:13:13.295 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:13:13.295 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:13.295 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:13:13.295 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:13.295 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:13.295 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-25 16:25:52.356139411 +0000 2024-03-25T18:13:13.295 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-25 16:17:58.030840896 +0000 2024-03-25T18:13:13.295 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-25 16:17:58.030840896 +0000 2024-03-25T18:13:13.296 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-25 16:17:58.030840896 +0000 2024-03-25T18:13:13.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:13:13.361 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-25T18:13:13.361 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-25T18:13:13.361 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.00021166 s, 2.4 MB/s 2024-03-25T18:13:13.362 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:13:13.418 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-25 16:25:52.357139390 +0000 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-25 16:17:58.364833559 +0000 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-25 16:17:58.364833559 +0000 2024-03-25T18:13:13.475 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-25 16:17:58.364833559 +0000 2024-03-25T18:13:13.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:13:13.540 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-25T18:13:13.540 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-25T18:13:13.540 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000240671 s, 2.1 MB/s 2024-03-25T18:13:13.541 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:13:13.597 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-25 16:25:52.357139390 +0000 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-25 16:17:58.728825562 +0000 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-25 16:17:58.728825562 +0000 2024-03-25T18:13:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-25 16:17:58.728825562 +0000 2024-03-25T18:13:13.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:13:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-25T18:13:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-25T18:13:13.716 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000193381 s, 2.6 MB/s 2024-03-25T18:13:13.717 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:13:13.773 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-03-25T18:13:13.828 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:13:13.828 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:13.829 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:13:13.829 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:13.829 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:13.829 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-03-25 16:25:52.357139390 +0000 2024-03-25T18:13:13.829 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-03-25 16:17:59.069818071 +0000 2024-03-25T18:13:13.829 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-03-25 16:17:59.069818071 +0000 2024-03-25T18:13:13.829 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-03-25 16:17:59.069818071 +0000 2024-03-25T18:13:13.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:13:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-03-25T18:13:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-03-25T18:13:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000193947 s, 2.6 MB/s 2024-03-25T18:13:13.895 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:13:13.951 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:13.951 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:13.951 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:13:13.967 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:13:13.967 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-03-25T18:13:14.022 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:13:14.022 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:14.022 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:13:14.022 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:14.022 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:14.023 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-25 16:22:26.431112315 +0000 2024-03-25T18:13:14.023 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-25 16:17:59.813485796 +0000 2024-03-25T18:13:14.023 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-25 16:17:59.813485796 +0000 2024-03-25T18:13:14.023 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-25 16:17:59.813485796 +0000 2024-03-25T18:13:14.023 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:13:14.089 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-25T18:13:14.089 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-25T18:13:14.089 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000275367 s, 1.9 MB/s 2024-03-25T18:13:14.090 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:13:14.147 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-25 16:22:26.431112315 +0000 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-25 16:18:00.160479527 +0000 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-25 16:18:00.160479527 +0000 2024-03-25T18:13:14.203 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-25 16:18:00.160479527 +0000 2024-03-25T18:13:14.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:13:14.268 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-25T18:13:14.268 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-25T18:13:14.268 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000213158 s, 2.4 MB/s 2024-03-25T18:13:14.269 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:13:14.325 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-03-25T18:13:14.380 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:13:14.380 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:14.380 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:13:14.380 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:14.380 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:14.380 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-25 16:22:26.432112298 +0000 2024-03-25T18:13:14.380 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-25 16:18:00.515473115 +0000 2024-03-25T18:13:14.380 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-25 16:18:00.515473115 +0000 2024-03-25T18:13:14.381 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-25 16:18:00.515473115 +0000 2024-03-25T18:13:14.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:13:14.445 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-25T18:13:14.446 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-25T18:13:14.446 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000187222 s, 2.7 MB/s 2024-03-25T18:13:14.447 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:13:14.502 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-03-25T18:13:14.557 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:13:14.557 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:14.557 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:13:14.557 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:14.557 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:14.557 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-03-25 16:22:26.432112298 +0000 2024-03-25T18:13:14.557 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-03-25 16:18:00.852467027 +0000 2024-03-25T18:13:14.557 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-03-25 16:18:00.852467027 +0000 2024-03-25T18:13:14.558 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-03-25 16:18:00.852467027 +0000 2024-03-25T18:13:14.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:13:14.623 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-03-25T18:13:14.623 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-03-25T18:13:14.623 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000205853 s, 2.5 MB/s 2024-03-25T18:13:14.624 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:13:14.680 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:14.680 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi162.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:13:14.680 INFO:tasks.ceph:Generating config... 2024-03-25T18:13:14.681 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:13:14.681 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:13:14.681 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:13:14.681 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:13:14.681 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:13:14.681 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:13:14.681 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:13:14.681 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:13:14.682 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:13:14.682 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:13:14.682 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:13:14.682 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:13:14.683 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:13:14.684 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:13:14.684 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:13:14.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:14.744 INFO:teuthology.orchestra.run.smithi162.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:13:14.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:13:14.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:14.820 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.162:6789]'), ('mon.c', '[v1:172.21.15.162:6790]'), ('mon.b', '[v1:172.21.15.191:6789]')] 2024-03-25T18:13:14.820 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.162:6789],[v1:172.21.15.162:6790],[v1:172.21.15.191:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:13:14.821 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:14.821 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:13:14.874 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.162:6789]' --addv c '[v1:172.21.15.162:6790]' --addv b '[v1:172.21.15.191:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:monmaptool: generated fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:setting min_mon_release = quincy 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:epoch 0 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:min_mon_release 17 (quincy) 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:election_strategy: 3 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi162.stdout:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:13:14.962 INFO:teuthology.orchestra.run.smithi162.stdout:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:13:14.962 INFO:teuthology.orchestra.run.smithi162.stdout:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:13:14.962 INFO:teuthology.orchestra.run.smithi162.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:13:14.963 DEBUG:teuthology.orchestra.run.smithi162:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:13:14.977 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e0e649a4-ed35-4995-8603-bf4aa2230a59... 2024-03-25T18:13:14.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:13:15.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout:[global] 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: chdir = "" 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: auth supported = cephx 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: filestore xattr use omap = true 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: mon clock drift allowed = 1.000 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: osd crush chooseleaf type = 0 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: auth debug = true 2024-03-25T18:13:15.068 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on old message = true 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: ms die on bug = true 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: mon pg warn max object skew = 0 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: osd pool default size = 2 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd allow primary affinity = true 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd allow pg remap = true 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:13:15.069 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on no sortbitwise = false 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on too few osds = false 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon_allow_pool_size_one = true 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: osd default data pool replay window = 5 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon allow pool delete = true 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon cluster log file level = debug 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: debug asserts on shutdown = true 2024-03-25T18:13:15.070 INFO:teuthology.orchestra.run.smithi162.stdout: mon health detail to clog = false 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: mon host = "[v1:172.21.15.162:6789],[v1:172.21.15.162:6790],[v1:172.21.15.191:6789]" 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: mon client directed command retry = 5 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: mon election default strategy = 3 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: ms bind msgr2 = False 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: ms inject socket failures = 5000 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: ms type = async 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: osd heartbeat grace = 80 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: fsid = e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout:[osd] 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: osd journal size = 100 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub load threshold = 5.0 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: osd scrub max interval = 600 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.071 INFO:teuthology.orchestra.run.smithi162.stdout: osd recover clone overlap = true 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd recovery max chunk = 1048576 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug shutdown = true 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug op order = true 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify stray on activate = true 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug trim objects = true 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd open classes on start = true 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug pg log writeout = true 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: osd map max advance = 10 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: journal zero on create = true 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.072 INFO:teuthology.orchestra.run.smithi162.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: filestore apply finisher threads = 3 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: bdev debug aio = true 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug misdirected ops = true 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: bdev async discard = True 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: bdev enable discard = True 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore allocator = stupid 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore block size = 96636764160 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore fsck on mount = True 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: bluestore zero block detection = True 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: debug bluefs = 1 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: debug bluestore = 1 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: debug monc = 20 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-25T18:13:15.073 INFO:teuthology.orchestra.run.smithi162.stdout: debug osd = 20 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: debug rocksdb = 10 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd full ratio = 0.9 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd class default list = * 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd class load list = * 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd client watch timeout = 120 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify cached snaps = True 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd debug verify missing on start = True 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd fast shutdown = False 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd max markdown count = 1000 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd mclock override recovery settings = True 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd objectstore = bluestore 2024-03-25T18:13:15.074 INFO:teuthology.orchestra.run.smithi162.stdout: osd op queue = debug_random 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: osd op queue cut off = debug_random 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: osd shutdown pgref assert = True 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr] 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: debug mgr = 20 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: debug mon = 20 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: debug auth = 20 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: mgr/telemetry/nag = false 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout:[mon] 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-25T18:13:15.075 INFO:teuthology.orchestra.run.smithi162.stdout: debug mon = 20 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: debug paxos = 20 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: debug auth = 20 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: mon data avail warn = 5 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: mon mgr mkfs grace = 240 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd reporter subtree level = osd 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd prime pg temp = true 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:13:15.076 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: # 1m isn't quite enough 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon min osdmap epochs = 50 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon osd crush smoke test = False 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune interval = 2 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune min = 15 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon scrub interval = 300 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: mon warn on pool no app = False 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: paxos service trim min = 10 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout:[client] 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: rgw cache enabled = true 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: rgw enable ops log = true 2024-03-25T18:13:15.077 INFO:teuthology.orchestra.run.smithi162.stdout: rgw enable usage log = true 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout: debug monc = 20 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout: debug ms = 1 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout: debug objecter = 20 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout: debug rados = 20 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.a] 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.c] 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi162.stdout:[mon.b] 2024-03-25T18:13:15.078 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.079 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:13:15.080 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "[v1:172.21.15.162:6789],[v1:172.21.15.162:6790],[v1:172.21.15.191:6789]" 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: mon election default strategy = 3 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: ms bind msgr2 = False 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 5000 2024-03-25T18:13:15.081 INFO:teuthology.orchestra.run.smithi191.stdout: ms type = async 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd heartbeat grace = 80 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2024-03-25T18:13:15.082 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug trim objects = true 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.083 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: bdev async discard = True 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: bdev enable discard = True 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore allocator = stupid 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore zero block detection = True 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 1 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 1 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: debug monc = 20 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:13:15.084 INFO:teuthology.orchestra.run.smithi191.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd class default list = * 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd class load list = * 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd client watch timeout = 120 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify cached snaps = True 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify missing on start = True 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd fast shutdown = False 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd max markdown count = 1000 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock override recovery settings = True 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue = debug_random 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd op queue cut off = debug_random 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: osd shutdown pgref assert = True 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2024-03-25T18:13:15.085 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:15.086 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: mon min osdmap epochs = 50 2024-03-25T18:13:15.087 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd crush smoke test = False 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune interval = 2 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune min = 15 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: mon scrub interval = 300 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on pool no app = False 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: paxos service trim min = 10 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: debug monc = 20 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2024-03-25T18:13:15.088 INFO:teuthology.orchestra.run.smithi191.stdout: debug objecter = 20 2024-03-25T18:13:15.089 INFO:teuthology.orchestra.run.smithi191.stdout: debug rados = 20 2024-03-25T18:13:15.089 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2024-03-25T18:13:15.089 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2024-03-25T18:13:15.089 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2024-03-25T18:13:15.090 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:13:15.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:13:15.145 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:13:15.145 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:15.145 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:13:15.159 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:15.159 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:13:15.214 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:13:15.214 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:15.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:13:15.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:15.293 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:15.293 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:15.348 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:13:15.348 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:15.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:13:15.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:15.388 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:15.388 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:15.443 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:13:15.443 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:13:15.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:13:15.498 INFO:teuthology.orchestra.run.smithi162.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:13:15.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:13:15.564 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:13:15.567 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:13:15.567 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:13:15.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:13:15.609 INFO:teuthology.orchestra.run.smithi162.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:13:15.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:13:15.668 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:13:15.683 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:13:15.683 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi162.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:13:15.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:15.709 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:15.709 INFO:tasks.ceph:role: osd.0 2024-03-25T18:13:15.710 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:13:15.710 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:15.784 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:16.291 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-03-25T18:13:16.293 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-03-25T18:13:16.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:16.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:16.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:16.544 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:16.544 INFO:tasks.ceph:role: osd.1 2024-03-25T18:13:16.545 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:13:16.545 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:13:16.619 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:16.619 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:16.619 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:16.619 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:16.619 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:16.619 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:16.620 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:16.620 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:16.620 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:16.620 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:17.063 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-03-25T18:13:17.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-03-25T18:13:17.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:17.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:17.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:17.255 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:17.255 INFO:tasks.ceph:role: osd.2 2024-03-25T18:13:17.255 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:13:17.255 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:13:17.330 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:17.330 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:17.330 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:17.330 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:17.330 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:17.331 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:17.331 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:17.331 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:17.331 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:17.331 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:17.816 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-03-25T18:13:17.819 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-03-25T18:13:17.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:17.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:17.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:18.047 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:18.047 INFO:tasks.ceph:role: osd.3 2024-03-25T18:13:18.047 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:13:18.048 DEBUG:teuthology.orchestra.run.smithi162:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:18.122 INFO:teuthology.orchestra.run.smithi162.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:18.588 INFO:teuthology.orchestra.run.smithi162.stdout:Discarding blocks...Done. 2024-03-25T18:13:18.590 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi162.front.sepia.ceph.com -o noatime 2024-03-25T18:13:18.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:18.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:18.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:18.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:18.846+0000 7f5fcd810640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:13:18.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:18.847+0000 7f5fcd810640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:13:18.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:18.847+0000 7f5fcd810640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:18.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:18.847+0000 7f5fcd810640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:18.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:18.847+0000 7f5fcd810640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:18.848 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:18.847+0000 7f5fcd810640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:13:21.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:21.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:21.827 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:21.826+0000 7f1dd71dd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:13:21.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:21.827+0000 7f1dd71dd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:13:21.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:21.827+0000 7f1dd71dd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:21.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:21.827+0000 7f1dd71dd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:21.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:21.827+0000 7f1dd71dd640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:21.828 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:21.827+0000 7f1dd71dd640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:13:24.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:24.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:24.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:24.786+0000 7f4575d49640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:13:24.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:24.787+0000 7f4575d49640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:13:24.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:24.787+0000 7f4575d49640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:24.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:24.787+0000 7f4575d49640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:24.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:24.787+0000 7f4575d49640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:24.788 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:24.787+0000 7f4575d49640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:13:27.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:27.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:27.758 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:27.758+0000 7fe68221c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:13:27.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:27.758+0000 7fe68221c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:13:27.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:27.758+0000 7fe68221c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:27.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:27.758+0000 7fe68221c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:27.759 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:27.758+0000 7fe68221c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:27.760 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:27.759+0000 7fe68221c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:13:30.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:30.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:30.664 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:30.665 INFO:tasks.ceph:role: osd.4 2024-03-25T18:13:30.665 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:13:30.665 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:13:30.742 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:30.742 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:30.743 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:30.743 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:30.743 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:30.743 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:30.743 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:30.743 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:30.743 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:30.743 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:31.253 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-25T18:13:31.256 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-25T18:13:31.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:31.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:31.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:31.515 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:31.515 INFO:tasks.ceph:role: osd.5 2024-03-25T18:13:31.515 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:13:31.515 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:31.592 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:32.038 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-25T18:13:32.041 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-25T18:13:32.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:32.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:32.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:32.242 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:32.242 INFO:tasks.ceph:role: osd.6 2024-03-25T18:13:32.242 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:13:32.242 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:13:32.318 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:32.318 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:32.318 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:32.319 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:32.319 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:32.319 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:32.319 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:32.319 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:32.319 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:32.319 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:32.807 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-25T18:13:32.809 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-25T18:13:32.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:32.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:32.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:33.056 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:33.056 INFO:tasks.ceph:role: osd.7 2024-03-25T18:13:33.056 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:13:33.056 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:13:33.132 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:33.133 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:33.630 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2024-03-25T18:13:33.632 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2024-03-25T18:13:33.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:33.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:33.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:33.899 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:33.898+0000 7f6e92c24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:13:33.899 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:33.898+0000 7f6e92c24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:13:33.899 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:33.898+0000 7f6e92c24640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:33.900 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:33.898+0000 7f6e92c24640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:33.900 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:33.898+0000 7f6e92c24640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:33.900 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:33.899+0000 7f6e92c24640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:13:36.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:36.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:36.869 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:36.869+0000 7fd52e215640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:13:36.870 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:36.869+0000 7fd52e215640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:13:36.870 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:36.869+0000 7fd52e215640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:36.870 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:36.869+0000 7fd52e215640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:36.870 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:36.869+0000 7fd52e215640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:36.871 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:36.870+0000 7fd52e215640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:13:39.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:39.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:39.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:39.834+0000 7fa12451f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:13:39.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:39.834+0000 7fa12451f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:13:39.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:39.834+0000 7fa12451f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:39.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:39.834+0000 7fa12451f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:39.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:39.834+0000 7fa12451f640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:39.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:39.834+0000 7fa12451f640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:13:42.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:42.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:42.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:42.797+0000 7faf8e506640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:13:42.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:42.797+0000 7faf8e506640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:13:42.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:42.797+0000 7faf8e506640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:42.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:42.797+0000 7faf8e506640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:42.798 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:42.797+0000 7faf8e506640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:42.799 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:13:42.798+0000 7faf8e506640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:13:45.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:45.665 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:13:45.666 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:45.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:13:45.698 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:45.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:13:45.768 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:45.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:13:45.838 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:45.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:13:45.906 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:45.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:13:45.975 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:45.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:13:46.007 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:46.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:13:46.077 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:46.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:13:46.146 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:46.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:13:46.215 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:46.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:13:46.283 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:13:46.283 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:13:46.300 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:13:46.300 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:13:46.339 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:13:46.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:13:46.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr.y] 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQC7vgFmh2a7HRAA1DWOlxM/69G8m92HY5YnYg== 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.0] 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQC+vgFmlpaJMhAA0FXmYcySRSndeSmjK8FAsQ== 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.1] 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDBvgFmIuBWMRAAmd9NS89sQr9Uut6MBXH6Tg== 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.2] 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDEvgFmYq33LhAA8beiFJhyVUt7p/i2hisE0g== 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.3] 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDHvgFmcBVALRAAkjrSpsQdd509tFBasNf2Ww== 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout:[mgr.x] 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQC7vgFm+letIRAAKyfROAnNoSUoOwO1qZ3bXg== 2024-03-25T18:13:46.383 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.4] 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDNvgFmbzueNRAASEXE/jyJjs+S/KA9uG4orw== 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.5] 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDQvgFmXJXeMxAAhCZizlcRT1OGCduAkNyJkg== 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.6] 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDTvgFm9P/FMRAACoL42eKtGaUlRKBi2EhwRg== 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout:[osd.7] 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDWvgFmEdOULxAAWugYXAdWDdeKY7K2ooz5iA== 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout:[client.0] 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQC7vgFm8MtaJBAAcpAOOqJPVgzT1ICbm4fdxg== 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout:[client.1] 2024-03-25T18:13:46.384 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQC7vgFmC8rbJxAAEr2lpE3g71lYzn3awt/ljg== 2024-03-25T18:13:46.405 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2024-03-25T18:13:46.405 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC7vgFmh2a7HRAA1DWOlxM/69G8m92HY5YnYg== 2024-03-25T18:13:46.405 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2024-03-25T18:13:46.405 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC+vgFmlpaJMhAA0FXmYcySRSndeSmjK8FAsQ== 2024-03-25T18:13:46.405 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2024-03-25T18:13:46.405 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDBvgFmIuBWMRAAmd9NS89sQr9Uut6MBXH6Tg== 2024-03-25T18:13:46.405 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDEvgFmYq33LhAA8beiFJhyVUt7p/i2hisE0g== 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDHvgFmcBVALRAAkjrSpsQdd509tFBasNf2Ww== 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC7vgFm+letIRAAKyfROAnNoSUoOwO1qZ3bXg== 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDNvgFmbzueNRAASEXE/jyJjs+S/KA9uG4orw== 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDQvgFmXJXeMxAAhCZizlcRT1OGCduAkNyJkg== 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDTvgFm9P/FMRAACoL42eKtGaUlRKBi2EhwRg== 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDWvgFmEdOULxAAWugYXAdWDdeKY7K2ooz5iA== 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2024-03-25T18:13:46.406 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC7vgFm8MtaJBAAcpAOOqJPVgzT1ICbm4fdxg== 2024-03-25T18:13:46.407 INFO:teuthology.orchestra.run.smithi191.stdout:[client.1] 2024-03-25T18:13:46.407 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQC7vgFmC8rbJxAAEr2lpE3g71lYzn3awt/ljg== 2024-03-25T18:13:46.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:46.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:46.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:46.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:46.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.876 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:46.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:46.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:47.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:47.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:47.092 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:13:47.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:13:47.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:47.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:13:47.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:13:47.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:47.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:13:47.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:13:47.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:47.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:13:47.583 DEBUG:teuthology.orchestra.run.smithi162:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:47.585 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:47.638 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:13:47.638 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:47.638 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:13:47.638 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:13:47.641 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:13:47.641 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:47.641 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:13:47.641 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:13:47.643 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:13:47.643 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:47.643 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:13:47.643 DEBUG:teuthology.orchestra.run.smithi191:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:13:47.680 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:13:47.680 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:13:47.680 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:13:47.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:13:47.683 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:13:47.683 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:13:47.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:13:47.684 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:13:47.685 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:13:47.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.786+0000 7f934e293640 10 monclient: get_monmap_and_config 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.786+0000 7f934e293640 10 monclient: build_initial_monmap 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.786+0000 7f934e293640 10 monclient: monmap: 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:47.787253+0000 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:47.787253+0000 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:13:47.787 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.786+0000 7f934e293640 1 Processor -- start 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 1 -- start start 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 10 monclient: init 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 10 monclient(hunting): picked mon.noname-a con 0x7f93480a9920 addr v1:172.21.15.162:6789/0 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 10 monclient(hunting): picked mon.noname-b con 0x7f93480b45a0 addr v1:172.21.15.162:6790/0 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 10 monclient(hunting): picked mon.noname-c con 0x7f9348146450 addr v1:172.21.15.191:6789/0 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934806b100 con 0x7f93480a9920 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480b2940 con 0x7f9348146450 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934813fec0 con 0x7f93480b45a0 2024-03-25T18:13:47.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:47.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934e293640 10 monclient(hunting): authenticate will time out at 7527.061339s 2024-03-25T18:13:47.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f934813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:47.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f9348145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:47.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.787+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f934814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:47.988 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.988+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f934813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:47.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.988+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f9348145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:47.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:47.988+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f934814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:48.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:48.389+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f934813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:48.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:48.389+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f9348145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:48.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:48.389+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f934814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:49.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:49.190+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f9348145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:49.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:49.190+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f934813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:49.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:49.190+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f934814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 10 monclient(hunting): tick 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f9348145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f934814dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f934813f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93480b45a0 addr v1:172.21.15.162:6790/0 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f93480a9920 addr v1:172.21.15.162:6789/0 2024-03-25T18:13:50.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9348146450 addr v1:172.21.15.191:6789/0 2024-03-25T18:13:50.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934806b100 con 0x7f93480a9920 2024-03-25T18:13:50.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480b2940 con 0x7f9348146450 2024-03-25T18:13:50.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:50.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934813fec0 con 0x7f93480b45a0 2024-03-25T18:13:50.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f9345ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:50.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f9334008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:50.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.787+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:50.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.988+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f9334008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:50.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.988+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:50.989 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:50.988+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:51.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:51.389+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:51.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:51.389+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f9334008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:51.390 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:51.389+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:52.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:52.190+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:52.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:52.190+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f9334008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:52.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:52.190+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:53.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.787+0000 7f9345ffb640 10 monclient(hunting): tick 2024-03-25T18:13:53.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.787+0000 7f9345ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:53.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.787+0000 7f9345ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:53.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.787+0000 7f9345ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:53.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.787+0000 7f9345ffb640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.787+0000 7f9345ffb640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.787+0000 7f9345ffb640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f9334008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9345ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f93480a9920 addr v1:172.21.15.162:6789/0 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93480b45a0 addr v1:172.21.15.162:6790/0 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9348146450 addr v1:172.21.15.191:6789/0 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9345ffb640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334010e20 con 0x7f93480a9920 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9345ffb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934806b100 con 0x7f9348146450 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9345ffb640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480b2940 con 0x7f93480b45a0 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9345ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f93340095d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:53.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.788+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:53.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.989+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f93340095d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:53.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.989+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:53.990 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:53.989+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:54.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:54.390+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f93340095d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:54.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:54.390+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:54.391 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:54.390+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:54.642 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:00:06.859 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:54.805 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:00:07.033 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:54.815 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:00:07.043 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:54.948 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:00:07.164 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:55.125 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:00:07.353 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:55.136 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:00:07.364 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:55.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:55.191+0000 7f93477fe640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6790/0 2024-03-25T18:13:55.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:55.191+0000 7f9347fff640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f93340095d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.162:6789/0 2024-03-25T18:13:55.191 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:55.191+0000 7f934c830640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.191:6789/0 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 10 monclient(hunting): tick 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 1 -- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:56.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 1 -- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f933400c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 1 -- >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f93340095d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f93480a9920 addr v1:172.21.15.162:6789/0 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9348146450 addr v1:172.21.15.191:6789/0 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93480b45a0 addr v1:172.21.15.162:6790/0 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934813fec0 con 0x7f93480a9920 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334010e20 con 0x7f9348146450 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934806b100 con 0x7f93480b45a0 2024-03-25T18:13:56.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.788+0000 7f9345ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.833+0000 7f934c830640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 0x7f93340056d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:60942/0 (socket says 172.21.15.162:60942) 2024-03-25T18:13:56.834 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:56.833+0000 7f934c830640 1 -- 172.21.15.162:0/1360650580 learned_addr learned my addr 172.21.15.162:0/1360650580 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:13:59.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 10 monclient(hunting): tick 2024-03-25T18:13:59.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:59.789 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f93340056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f93340095f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.162:6789/0 conn(0x7f93480a9920 legacy=0x7f9334008e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f933400c740 addr v1:172.21.15.162:6789/0 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9348146450 addr v1:172.21.15.191:6789/0 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93480b45a0 addr v1:172.21.15.162:6790/0 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93480b2940 con 0x7f933400c740 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340066e0 con 0x7f9348146450 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334007880 con 0x7f93480b45a0 2024-03-25T18:13:59.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:13:59.788+0000 7f9345ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 10 monclient(hunting): tick 2024-03-25T18:14:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f9334008e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f93340095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.162:6789/0 conn(0x7f933400c740 legacy=0x7f9334009da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:02.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f9334007a10 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:02.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.788+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f93480b45a0 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:02.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.789+0000 7f9345ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f9348146450 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:02.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.789+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334008a70 con 0x7f9334007a10 2024-03-25T18:14:02.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.789+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334008560 con 0x7f9348146450 2024-03-25T18:14:02.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.789+0000 7f9345ffb640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9334008740 con 0x7f93480b45a0 2024-03-25T18:14:02.791 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:02.789+0000 7f9345ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:05.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.514+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419799398 0 0) 0x7f9334008a70 con 0x7f9334007a10 2024-03-25T18:14:05.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.515+0000 7f9346ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:05.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.515+0000 7f9346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:05.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.515+0000 7f9346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:05.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.515+0000 7f9346ffd640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:14:05.516 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.515+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9330003520 con 0x7f9334007a10 2024-03-25T18:14:05.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.518+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073819091 0 0) 0x7f9334008740 con 0x7f93480b45a0 2024-03-25T18:14:05.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.518+0000 7f9346ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:05.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.518+0000 7f9346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:05.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.518+0000 7f9346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:05.519 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.518+0000 7f9346ffd640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:14:05.520 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.518+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9334008a70 con 0x7f93480b45a0 2024-03-25T18:14:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.533+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532089878 0 0) 0x7f9334008560 con 0x7f9348146450 2024-03-25T18:14:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.533+0000 7f9346ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.533+0000 7f9346ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.533+0000 7f9346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.533+0000 7f9346ffd640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:14:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.533+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9334008740 con 0x7f9348146450 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 922652947 0 0) 0x7f9330003520 con 0x7f9334007a10 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.162:6790/0 conn(0x7f93480b45a0 legacy=0x7f9334008e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f93340095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 1 monclient: found mon.noname-a 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334009da0 con 0x7f9334007a10 2024-03-25T18:14:05.541 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9338004080 con 0x7f9334007a10 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f9346ffd640 10 monclient: dump: 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:05.543 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:05.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f934e293640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:14:05.543 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.540+0000 7f934e293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:05.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.581+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93380041e0 con 0x7f9334007a10 2024-03-25T18:14:05.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.581+0000 7f9346ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:05.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.581+0000 7f9346ffd640 1 -- 172.21.15.162:0/1360650580 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9338004630 con 0x7f9334007a10 2024-03-25T18:14:05.582 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.581+0000 7f9346ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.581+0000 7f9346ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.581+0000 7f9346ffd640 10 monclient: dump: 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.581+0000 7f934e293640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.581+0000 7f934e293640 4 set_mon_vals no callback set 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.582+0000 7f934e293640 10 monclient: shutdown 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.582+0000 7f934e293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.582+0000 7f934e293640 1 -- 172.21.15.162:0/1360650580 >> v1:172.21.15.162:6789/0 conn(0x7f9334007a10 legacy=0x7f9334007e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.582+0000 7f934e293640 1 -- 172.21.15.162:0/1360650580 shutdown_connections 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.582+0000 7f934e293640 1 -- 172.21.15.162:0/1360650580 >> 172.21.15.162:0/1360650580 conn(0x7f93480b7f40 msgr2=0x7f93480b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.582+0000 7f934e293640 1 -- 172.21.15.162:0/1360650580 shutdown_connections 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.582+0000 7f934e293640 1 -- 172.21.15.162:0/1360650580 wait complete. 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.583+0000 7f934e293640 10 monclient: build_initial_monmap 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.583+0000 7f934e293640 10 monclient: monmap: 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-03-25T18:14:05.586 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.583+0000 7f934e293640 1 librados: starting msgr at 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.583+0000 7f934e293640 1 librados: starting objecter 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.583+0000 7f934e293640 1 Processor -- start 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 1 -- start start 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 1 librados: setting wanted keys 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 1 librados: calling monclient init 2024-03-25T18:14:05.587 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 10 monclient: init 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 10 monclient(hunting): picked mon.noname-c con 0x7f9334007a10 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 10 monclient(hunting): picked mon.noname-a con 0x7f93480b45a0 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 10 monclient(hunting): picked mon.noname-b con 0x7f9348146450 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93481433d0 con 0x7f93480b45a0 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93481453c0 con 0x7f9348146450 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9348136aa0 con 0x7f9334007a10 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.584+0000 7f934e293640 10 monclient(hunting): authenticate will time out at 7544.858337s 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.585+0000 7f934c830640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 0x7f9348136390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:34730/0 (socket says 172.21.15.162:34730) 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.585+0000 7f934c830640 1 -- 172.21.15.162:0/2828846544 learned_addr learned my addr 172.21.15.162:0/2828846544 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.585+0000 7f9344ff9640 10 client.?.objecter ms_handle_connect 0x7f9348146450 2024-03-25T18:14:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.585+0000 7f9344ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:05.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.587+0000 7f9344ff9640 10 client.?.objecter ms_handle_connect 0x7f9334007a10 2024-03-25T18:14:05.589 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.587+0000 7f9344ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:05.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.597+0000 7f9344ff9640 10 client.?.objecter ms_handle_connect 0x7f93480b45a0 2024-03-25T18:14:05.598 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.597+0000 7f9344ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:05.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.650+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511600936 0 0) 0x7f93481433d0 con 0x7f93480b45a0 2024-03-25T18:14:05.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.650+0000 7f9344ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:05.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.650+0000 7f9344ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:05.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.650+0000 7f9344ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:05.651 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.650+0000 7f9344ff9640 10 monclient(hunting): my global_id is 4110 2024-03-25T18:14:05.652 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.650+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93340033f0 con 0x7f93480b45a0 2024-03-25T18:14:05.678 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.678+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353348285 0 0) 0x7f9348136aa0 con 0x7f9334007a10 2024-03-25T18:14:05.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.678+0000 7f9344ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:05.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.678+0000 7f9344ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:05.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.678+0000 7f9344ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:05.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.678+0000 7f9344ff9640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:14:05.679 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.678+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93481433d0 con 0x7f9334007a10 2024-03-25T18:14:05.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.699+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 162167469 0 0) 0x7f93340033f0 con 0x7f93480b45a0 2024-03-25T18:14:05.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.699+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9348136aa0 con 0x7f93480b45a0 2024-03-25T18:14:05.700 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.699+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f933c004200 con 0x7f93480b45a0 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.699+0000 7f9344ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.699+0000 7f9344ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.699+0000 7f9344ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:05.701 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.699+0000 7f9344ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:05.727 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1011848294 0 0) 0x7f9348136aa0 con 0x7f93480b45a0 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 >> v1:172.21.15.162:6790/0 conn(0x7f9334007a10 legacy=0x7f9348143860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 >> v1:172.21.15.191:6789/0 conn(0x7f9348146450 legacy=0x7f9348136390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 1 monclient: found mon.a 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9348145630 con 0x7f93480b45a0 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f9344ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 5 monclient: authenticate success, global_id 4110 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 10 monclient: _renew_subs 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:05.728 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9348142f60 con 0x7f93480b45a0 2024-03-25T18:14:05.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:05.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 10 monclient: _renew_subs 2024-03-25T18:14:05.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:05.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93481db9e0 con 0x7f93480b45a0 2024-03-25T18:14:05.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.726+0000 7f934e293640 1 librados: init done 2024-03-25T18:14:05.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.727+0000 7f934e293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:05.729 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.728+0000 7f934e293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:05.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.728+0000 7f934e293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:05.730 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.728+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9348143580 con 0x7f93480b45a0 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f933c004360 con 0x7f93480b45a0 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f933c004590 con 0x7f93480b45a0 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 10 monclient: dump: 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f933c004b40 con 0x7f93480b45a0 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f93467fc640 4 set_mon_vals no callback set 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2388180129 0 0) 0x7f933c006340 con 0x7f93480b45a0 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 10 client.4110.objecter ms_dispatch 0x7f9348000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.909+0000 7f9344ff9640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.910+0000 7f9344ff9640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.910+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f933c006780 con 0x7f93480b45a0 2024-03-25T18:14:05.912 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.910+0000 7f9344ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:05.913 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.910+0000 7f9344ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:05.983 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.982+0000 7f934e293640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:05.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.982+0000 7f934e293640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:05.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.982+0000 7f934e293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:05.984 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:05.982+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f934804f730 con 0x7f93480b45a0 2024-03-25T18:14:06.166 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:06.166+0000 7f214ce32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:14:06.211 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.210+0000 7f9344ff9640 1 -- 172.21.15.162:0/2828846544 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f933c004ed0 con 0x7f93480b45a0 2024-03-25T18:14:06.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.211+0000 7f9344ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:06.212 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.211+0000 7f9344ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:14:06.212 INFO:teuthology.orchestra.run.smithi162.stderr:adjusted tunables profile to default 2024-03-25T18:14:06.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 10 librados: watch_flush enter 2024-03-25T18:14:06.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 10 librados: watch_flush exit 2024-03-25T18:14:06.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-25T18:14:06.213 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 10 client.4110.objecter successfully canceled tick 2024-03-25T18:14:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 10 monclient: shutdown 2024-03-25T18:14:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 >> v1:172.21.15.162:6789/0 conn(0x7f93480b45a0 legacy=0x7f9348135c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 shutdown_connections 2024-03-25T18:14:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 >> 172.21.15.162:0/2828846544 conn(0x7f93480b7f40 msgr2=0x7f934813e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 shutdown_connections 2024-03-25T18:14:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 1 -- 172.21.15.162:0/2828846544 wait complete. 2024-03-25T18:14:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:06.212+0000 7f934e293640 1 librados: shutdown 2024-03-25T18:14:06.225 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:14:06.225 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:14:06.225 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:14:06.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:14:06.225 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:06.224+0000 7f214ce32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:14:06.253 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:14:06.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:14:06.322 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:14:06.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:14:06.389 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-25T18:14:06.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:14:06.389 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:06.389+0000 7f49ca04c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:14:06.445 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:06.444+0000 7f49ca04c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:14:06.456 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:14:06.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:14:06.483 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:14:06.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:14:06.550 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:14:06.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:14:06.618 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-03-25T18:14:06.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:14:06.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 6f567bd1-d9ae-41cb-b745-c7a607ecf767 0 2024-03-25T18:14:06.812 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:06.811+0000 7f214ce32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.824+0000 7fb867633640 10 monclient: get_monmap_and_config 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.824+0000 7fb867633640 10 monclient: build_initial_monmap 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.824+0000 7fb867633640 10 monclient: monmap: 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:14:06.825075+0000 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:14:06.825075+0000 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:06.825 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:06.826 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:06.826 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:06.826 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:06.826 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:06.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.824+0000 7fb867633640 1 Processor -- start 2024-03-25T18:14:06.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.824+0000 7fb867633640 1 -- start start 2024-03-25T18:14:06.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.824+0000 7fb867633640 10 monclient: init 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 10 monclient(hunting): picked mon.noname-b con 0x7fb8600b4700 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 10 monclient(hunting): picked mon.noname-a con 0x7fb860142a20 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 10 monclient(hunting): picked mon.noname-c con 0x7fb8600a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8600a9d60 con 0x7fb860142a20 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86013ff10 con 0x7fb8600a96b0 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8600aa1f0 con 0x7fb8600b4700 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.825+0000 7fb867633640 10 monclient(hunting): authenticate will time out at 7529.160294s 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.826+0000 7fb8653cf640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fb8600b4700 0x7fb860142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.191:42008/0 (socket says 172.21.15.191:42008) 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.826+0000 7fb8653cf640 1 -- 172.21.15.191:0/3285327390 learned_addr learned my addr 172.21.15.191:0/3285327390 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:06.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.826+0000 7fb864bce640 1 --1- 172.21.15.191:0/3285327390 >> v1:172.21.15.162:6789/0 conn(0x7fb860142a20 0x7fb86013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.191:50832/0 (socket says 172.21.15.191:50832) 2024-03-25T18:14:06.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687636238 0 0) 0x7fb8600a9d60 con 0x7fb860142a20 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c003520 con 0x7fb860142a20 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397652565 0 0) 0x7fb86013ff10 con 0x7fb8600a96b0 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8600a9d60 con 0x7fb8600a96b0 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930776761 0 0) 0x7fb8600aa1f0 con 0x7fb8600b4700 2024-03-25T18:14:06.830 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:14:06.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.828+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86013ff10 con 0x7fb8600b4700 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852507990 0 0) 0x7fb84c003520 con 0x7fb860142a20 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 >> v1:172.21.15.162:6790/0 conn(0x7fb8600b4700 legacy=0x7fb860142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 >> v1:172.21.15.191:6789/0 conn(0x7fb8600a96b0 legacy=0x7fb86013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 1 monclient: found mon.noname-a 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8600b4ae0 con 0x7fb860142a20 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:06.833 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.831+0000 7fb857fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.832+0000 7fb867633640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.832+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb85c001690 con 0x7fb860142a20 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.832+0000 7fb867633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.832+0000 7fb857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.832+0000 7fb857fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.832+0000 7fb857fff640 10 monclient: dump: 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:06.834 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:06.835 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:06.835 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:06.835 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:06.835 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:06.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.832+0000 7fb867633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.833+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb85c0017f0 con 0x7fb860142a20 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.833+0000 7fb857fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.833+0000 7fb857fff640 1 -- 172.21.15.191:0/3285327390 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb85c001c00 con 0x7fb860142a20 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.833+0000 7fb867633640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.833+0000 7fb867633640 4 set_mon_vals no callback set 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.833+0000 7fb857fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.833+0000 7fb857fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.833+0000 7fb857fff640 10 monclient: dump: 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:06.838 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.834+0000 7fb867633640 10 monclient: shutdown 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.834+0000 7fb867633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.834+0000 7fb867633640 1 -- 172.21.15.191:0/3285327390 >> v1:172.21.15.162:6789/0 conn(0x7fb860142a20 legacy=0x7fb86013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.834+0000 7fb867633640 1 -- 172.21.15.191:0/3285327390 shutdown_connections 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.834+0000 7fb867633640 1 -- 172.21.15.191:0/3285327390 >> 172.21.15.191:0/3285327390 conn(0x7fb8600668e0 msgr2=0x7fb860066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.834+0000 7fb867633640 1 -- 172.21.15.191:0/3285327390 shutdown_connections 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.834+0000 7fb867633640 1 -- 172.21.15.191:0/3285327390 wait complete. 2024-03-25T18:14:06.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 10 monclient: build_initial_monmap 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 10 monclient: monmap: 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 1 librados: starting msgr at 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 1 librados: starting objecter 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 1 Processor -- start 2024-03-25T18:14:06.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 1 -- start start 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 1 librados: setting wanted keys 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 1 librados: calling monclient init 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.835+0000 7fb867633640 10 monclient: init 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 10 monclient(hunting): picked mon.noname-a con 0x7fb8600a96b0 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 10 monclient(hunting): picked mon.noname-c con 0x7fb8600b4700 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 10 monclient(hunting): picked mon.noname-b con 0x7fb860142a20 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8600ac680 con 0x7fb8600a96b0 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86014e740 con 0x7fb860142a20 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86014eb90 con 0x7fb8600b4700 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb867633640 10 monclient(hunting): authenticate will time out at 7529.170951s 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb8653cf640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb8600a96b0 0x7fb86013ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.191:50844/0 (socket says 172.21.15.191:50844) 2024-03-25T18:14:06.841 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb8653cf640 1 -- 172.21.15.191:0/3378686916 learned_addr learned my addr 172.21.15.191:0/3378686916 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:06.842 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb855ffb640 10 client.?.objecter ms_handle_connect 0x7fb8600a96b0 2024-03-25T18:14:06.842 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.836+0000 7fb855ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:06.842 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.839+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525080637 0 0) 0x7fb8600ac680 con 0x7fb8600a96b0 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.839+0000 7fb855ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.839+0000 7fb855ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.839+0000 7fb855ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.839+0000 7fb855ffb640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.839+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb844004eb0 con 0x7fb8600a96b0 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.839+0000 7fb855ffb640 10 client.?.objecter ms_handle_connect 0x7fb8600b4700 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.839+0000 7fb855ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.841+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481172961 0 0) 0x7fb86014eb90 con 0x7fb8600b4700 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.841+0000 7fb855ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.841+0000 7fb855ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.841+0000 7fb855ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:06.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.841+0000 7fb855ffb640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:14:06.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.841+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8600ac680 con 0x7fb8600b4700 2024-03-25T18:14:06.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.842+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3634210864 0 0) 0x7fb844004eb0 con 0x7fb8600a96b0 2024-03-25T18:14:06.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.842+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb86014eb90 con 0x7fb8600a96b0 2024-03-25T18:14:06.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.842+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb850001910 con 0x7fb8600a96b0 2024-03-25T18:14:06.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.842+0000 7fb855ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.842+0000 7fb855ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.842+0000 7fb855ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:06.845 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.842+0000 7fb855ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:06.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1742593723 0 0) 0x7fb86014eb90 con 0x7fb8600a96b0 2024-03-25T18:14:06.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 >> v1:172.21.15.162:6790/0 conn(0x7fb8600b4700 legacy=0x7fb86013c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 >> v1:172.21.15.191:6789/0 conn(0x7fb860142a20 legacy=0x7fb860139dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 1 monclient: found mon.a 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8600b1490 con 0x7fb8600a96b0 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb855ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 10 monclient: _renew_subs 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8601e3090 con 0x7fb8600a96b0 2024-03-25T18:14:06.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:06.850 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 10 monclient: _renew_subs 2024-03-25T18:14:06.850 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:06.850 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8601e3400 con 0x7fb8600a96b0 2024-03-25T18:14:06.850 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.847+0000 7fb867633640 1 librados: init done 2024-03-25T18:14:06.850 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.848+0000 7fb867633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:06.850 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.848+0000 7fb867633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:06.850 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.848+0000 7fb867633640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:06.851 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.848+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8600b16e0 con 0x7fb8600a96b0 2024-03-25T18:14:06.871 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:06.870+0000 7f214ce32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:14:06.918 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:06.917+0000 7f214ce32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.952+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8500047e0 con 0x7fb8600a96b0 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.952+0000 7fb855ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.952+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb850004bf0 con 0x7fb8600a96b0 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.952+0000 7fb855ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.952+0000 7fb855ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.952+0000 7fb855ffb640 10 monclient: dump: 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:06.953 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:06.954 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:06.954 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:06.954 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:06.954 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:06.954 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:06.954 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:06.954 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.952+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb8500051a0 con 0x7fb8600a96b0 2024-03-25T18:14:06.954 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.952+0000 7fb8577fe640 4 set_mon_vals no callback set 2024-03-25T18:14:06.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.953+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 4026334007 0 0) 0x7fb8500065d0 con 0x7fb8600a96b0 2024-03-25T18:14:06.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.953+0000 7fb855ffb640 10 client.4119.objecter ms_dispatch 0x7fb860000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:14:06.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.953+0000 7fb855ffb640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:14:06.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.953+0000 7fb855ffb640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:14:06.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.953+0000 7fb855ffb640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:14:06.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.953+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb850006920 con 0x7fb8600a96b0 2024-03-25T18:14:06.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.953+0000 7fb855ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:06.956 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:06.953+0000 7fb855ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:06.973 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:06.973+0000 7f214ce32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:14:06.985 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:06.984+0000 7f49ca04c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:14:07.016 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:07.015+0000 7f214ce32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:14:07.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.025+0000 7fb867633640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6f567bd1-d9ae-41cb-b745-c7a607ecf767", "id": 0}] 2024-03-25T18:14:07.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.025+0000 7fb867633640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6f567bd1-d9ae-41cb-b745-c7a607ecf767", "id": 0}] 2024-03-25T18:14:07.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.025+0000 7fb867633640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:07.026 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.025+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6f567bd1-d9ae-41cb-b745-c7a607ecf767", "id": 0} v 0) v1 -- 0x7fb8600ac4a0 con 0x7fb8600a96b0 2024-03-25T18:14:07.040 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.039+0000 7f49ca04c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:14:07.084 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.083+0000 7f49ca04c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:14:07.135 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.134+0000 7fb855ffb640 1 -- 172.21.15.191:0/3378686916 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f567bd1-d9ae-41cb-b745-c7a607ecf767", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3513630971 0 732707515) 0x7fb850006ac0 con 0x7fb8600a96b0 2024-03-25T18:14:07.135 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.134+0000 7fb855ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6f567bd1-d9ae-41cb-b745-c7a607ecf767", "id": 0}] 2024-03-25T18:14:07.135 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.134+0000 7fb855ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:07.136 INFO:teuthology.orchestra.run.smithi191.stdout:0 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 10 librados: watch_flush enter 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 10 librados: watch_flush exit 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 10 monclient: shutdown 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 >> v1:172.21.15.162:6789/0 conn(0x7fb8600a96b0 legacy=0x7fb86013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 shutdown_connections 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 >> 172.21.15.191:0/3378686916 conn(0x7fb8600668e0 msgr2=0x7fb8600b6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 shutdown_connections 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 1 -- 172.21.15.191:0/3378686916 wait complete. 2024-03-25T18:14:07.137 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.135+0000 7fb867633640 1 librados: shutdown 2024-03-25T18:14:07.140 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:07.139+0000 7f214ce32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:14:07.142 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.141+0000 7f49ca04c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:14:07.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 450b2740-938f-45f6-96f9-7774f90ae10f 1 2024-03-25T18:14:07.191 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.190+0000 7f49ca04c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:14:07.212 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:07.211+0000 7f214ce32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:07.250 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.249+0000 7f225ed54640 10 monclient: get_monmap_and_config 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.249+0000 7f225ed54640 10 monclient: build_initial_monmap 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.249+0000 7f225ed54640 10 monclient: monmap: 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:14:07.250780+0000 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:14:07.250780+0000 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:07.251 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.252 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.250+0000 7f225ed54640 1 Processor -- start 2024-03-25T18:14:07.252 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.250+0000 7f225ed54640 1 -- start start 2024-03-25T18:14:07.252 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.250+0000 7f225ed54640 10 monclient: init 2024-03-25T18:14:07.252 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 10 monclient(hunting): picked mon.noname-c con 0x7f22580b4700 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 10 monclient(hunting): picked mon.noname-b con 0x7f2258142a20 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 10 monclient(hunting): picked mon.noname-a con 0x7f22580a96b0 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22580a9d60 con 0x7f22580a96b0 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225813ff10 con 0x7f22580b4700 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22580aa1f0 con 0x7f2258142a20 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225ed54640 10 monclient(hunting): authenticate will time out at 7529.585875s 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225d2f1640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f22580a96b0 0x7f225813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.191:50854/0 (socket says 172.21.15.191:50854) 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f225d2f1640 1 -- 172.21.15.191:0/243132048 learned_addr learned my addr 172.21.15.191:0/243132048 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:07.253 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.251+0000 7f2257fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f2258142a20 0x7f225813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.191:42032/0 (socket says 172.21.15.191:42032) 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574073822 0 0) 0x7f22580a9d60 con 0x7f22580a96b0 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2248003520 con 0x7f22580a96b0 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877996775 0 0) 0x7f22580aa1f0 con 0x7f2258142a20 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.255 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.256 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:14:07.256 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22580a9d60 con 0x7f2258142a20 2024-03-25T18:14:07.256 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414513810 0 0) 0x7f225813ff10 con 0x7f22580b4700 2024-03-25T18:14:07.256 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.256 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.256 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.256 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:14:07.256 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.254+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22580aa1f0 con 0x7f22580b4700 2024-03-25T18:14:07.258 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926779803 0 0) 0x7f22580a9d60 con 0x7f2258142a20 2024-03-25T18:14:07.258 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 >> v1:172.21.15.191:6789/0 conn(0x7f22580b4700 legacy=0x7f2258142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.258 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 >> v1:172.21.15.162:6789/0 conn(0x7f22580a96b0 legacy=0x7f225813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.258 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 1 monclient: found mon.noname-b 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22580b4ae0 con 0x7f2258142a20 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f225ed54640 5 monclient: authenticate success, global_id 4121 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f225ed54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2244001630 con 0x7f2258142a20 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f22577fe640 10 monclient: dump: 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.259 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.260 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.260 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.260 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.260 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.260 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.260 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.260 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.260 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.257+0000 7f225ed54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:07.260 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:07.259+0000 7f214ce32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:14:07.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2244001880 con 0x7f2258142a20 2024-03-25T18:14:07.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f22577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.284 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f22577fe640 1 -- 172.21.15.191:0/243132048 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2244001c10 con 0x7f2258142a20 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f22577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f22577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f22577fe640 10 monclient: dump: 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:07.285 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 4 set_mon_vals no callback set 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 10 monclient: shutdown 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 1 -- 172.21.15.191:0/243132048 >> v1:172.21.15.162:6790/0 conn(0x7f2258142a20 legacy=0x7f225813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 1 -- 172.21.15.191:0/243132048 shutdown_connections 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 1 -- 172.21.15.191:0/243132048 >> 172.21.15.191:0/243132048 conn(0x7f22580668e0 msgr2=0x7f2258066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 1 -- 172.21.15.191:0/243132048 shutdown_connections 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.284+0000 7f225ed54640 1 -- 172.21.15.191:0/243132048 wait complete. 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 10 monclient: build_initial_monmap 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 10 monclient: monmap: 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:07.288 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 1 librados: starting msgr at 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 1 librados: starting objecter 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 1 Processor -- start 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 1 -- start start 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 1 librados: setting wanted keys 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 1 librados: calling monclient init 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.285+0000 7f225ed54640 10 monclient: init 2024-03-25T18:14:07.289 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 10 monclient(hunting): picked mon.noname-b con 0x7f22580a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 10 monclient(hunting): picked mon.noname-c con 0x7f22580b4700 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 10 monclient(hunting): picked mon.noname-a con 0x7f2258142a20 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2258138960 con 0x7f2258142a20 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2258139bb0 con 0x7f22580a96b0 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f225813a650 con 0x7f22580b4700 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.286+0000 7f225ed54640 10 monclient(hunting): authenticate will time out at 7529.621425s 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.287+0000 7f2257fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f22580b4700 0x7f2258139f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.191:42046/0 (socket says 172.21.15.191:42046) 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.287+0000 7f2257fff640 1 -- 172.21.15.191:0/3622701139 learned_addr learned my addr 172.21.15.191:0/3622701139 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.287+0000 7f22557fa640 10 client.?.objecter ms_handle_connect 0x7f22580b4700 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.287+0000 7f22557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.287+0000 7f22557fa640 10 client.?.objecter ms_handle_connect 0x7f2258142a20 2024-03-25T18:14:07.291 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.287+0000 7f22557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.291 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.287+0000 7f22557fa640 10 client.?.objecter ms_handle_connect 0x7f22580a96b0 2024-03-25T18:14:07.291 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.287+0000 7f22557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.289+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932968767 0 0) 0x7f2258138960 con 0x7f2258142a20 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.289+0000 7f22557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.289+0000 7f22557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.289+0000 7f22557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.289+0000 7f22557fa640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.289+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c004eb0 con 0x7f2258142a20 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839503253 0 0) 0x7f2258139bb0 con 0x7f22580a96b0 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2258138960 con 0x7f22580a96b0 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072014388 0 0) 0x7f225813a650 con 0x7f22580b4700 2024-03-25T18:14:07.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:14:07.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.290+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2258139bb0 con 0x7f22580b4700 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4252283127 0 0) 0x7f2258139bb0 con 0x7f22580b4700 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f225813a650 con 0x7f22580b4700 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2244001b90 con 0x7f22580b4700 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 10 monclient(hunting): dump: 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.295 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2524076390 0 0) 0x7f223c004eb0 con 0x7f2258142a20 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2258139bb0 con 0x7f2258142a20 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2250001c50 con 0x7f2258142a20 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.294+0000 7f22557fa640 10 monclient(hunting): dump: 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.296 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.297 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.297 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.297 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.297 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.297 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.297 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.297 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1672482100 0 0) 0x7f2258139bb0 con 0x7f2258142a20 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 >> v1:172.21.15.162:6790/0 conn(0x7f22580b4700 legacy=0x7f2258139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 >> v1:172.21.15.191:6789/0 conn(0x7f22580a96b0 legacy=0x7f22581380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 1 monclient: found mon.a 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225813cd60 con 0x7f2258142a20 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f22557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.299 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:14:07.300 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 10 monclient: _renew_subs 2024-03-25T18:14:07.300 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:07.300 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2258136960 con 0x7f2258142a20 2024-03-25T18:14:07.300 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:07.300 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 10 monclient: _renew_subs 2024-03-25T18:14:07.300 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:07.300 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2258137700 con 0x7f2258142a20 2024-03-25T18:14:07.300 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 1 librados: init done 2024-03-25T18:14:07.301 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.301 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.301 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:07.301 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.298+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225804f730 con 0x7f2258142a20 2024-03-25T18:14:07.303 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2250002060 con 0x7f2258142a20 2024-03-25T18:14:07.303 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.303 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2250004d00 con 0x7f2258142a20 2024-03-25T18:14:07.303 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.303 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:07.303 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 10 monclient: dump: 2024-03-25T18:14:07.303 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.303 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f22500052b0 con 0x7f2258142a20 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 488385948 0 0) 0x7f2250006770 con 0x7f2258142a20 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 10 client.4125.objecter ms_dispatch 0x7f2258000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22567fc640 4 set_mon_vals no callback set 2024-03-25T18:14:07.304 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.302+0000 7f22557fa640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:14:07.314 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.314+0000 7f49ca04c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:14:07.317 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:07.317+0000 7f214ce32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:14:07.367 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:07.366+0000 7f214ce32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:14:07.384 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.383+0000 7f49ca04c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:07.409 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.408+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2250005c50 con 0x7f2258142a20 2024-03-25T18:14:07.409 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.408+0000 7f22557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.409 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.408+0000 7f22557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:07.432 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.432+0000 7f49ca04c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:14:07.485 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.484+0000 7f225ed54640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "450b2740-938f-45f6-96f9-7774f90ae10f", "id": 1}] 2024-03-25T18:14:07.485 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.484+0000 7f225ed54640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "450b2740-938f-45f6-96f9-7774f90ae10f", "id": 1}] 2024-03-25T18:14:07.485 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.484+0000 7f225ed54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:07.485 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.484+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd new", "uuid": "450b2740-938f-45f6-96f9-7774f90ae10f", "id": 1} v 0) v1 -- 0x7f22580b7030 con 0x7f2258142a20 2024-03-25T18:14:07.488 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.487+0000 7f49ca04c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:14:07.525 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.524+0000 7f22557fa640 1 -- 172.21.15.191:0/3622701139 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "450b2740-938f-45f6-96f9-7774f90ae10f", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3844937610 0 940483276) 0x7f2250005df0 con 0x7f2258142a20 2024-03-25T18:14:07.525 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.524+0000 7f22557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "450b2740-938f-45f6-96f9-7774f90ae10f", "id": 1}] 2024-03-25T18:14:07.525 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.524+0000 7f22557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:07.525 INFO:teuthology.orchestra.run.smithi191.stdout:1 2024-03-25T18:14:07.526 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 10 librados: watch_flush enter 2024-03-25T18:14:07.526 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 10 librados: watch_flush exit 2024-03-25T18:14:07.526 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:14:07.526 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:14:07.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 10 monclient: shutdown 2024-03-25T18:14:07.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:07.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 >> v1:172.21.15.162:6789/0 conn(0x7f2258142a20 legacy=0x7f22581e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 shutdown_connections 2024-03-25T18:14:07.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 >> 172.21.15.191:0/3622701139 conn(0x7f22580668e0 msgr2=0x7f225813f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:07.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 shutdown_connections 2024-03-25T18:14:07.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 1 -- 172.21.15.191:0/3622701139 wait complete. 2024-03-25T18:14:07.527 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.525+0000 7f225ed54640 1 librados: shutdown 2024-03-25T18:14:07.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 24966f74-30d0-4d45-9c63-61d635d94d26 2 2024-03-25T18:14:07.539 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.538+0000 7f49ca04c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.638+0000 7fc2d12a8640 10 monclient: get_monmap_and_config 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.638+0000 7fc2d12a8640 10 monclient: build_initial_monmap 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.638+0000 7fc2d12a8640 10 monclient: monmap: 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:14:07.639270+0000 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:14:07.639270+0000 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:07.639 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.640 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.638+0000 7fc2d12a8640 1 Processor -- start 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.638+0000 7fc2d12a8640 1 -- start start 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.638+0000 7fc2d12a8640 10 monclient: init 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc2cc0b4700 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc2cc142a20 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc2cc0a96b0 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc0a9d60 con 0x7fc2cc0b4700 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc13ff10 con 0x7fc2cc142a20 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc0aa1f0 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.639+0000 7fc2d12a8640 10 monclient(hunting): authenticate will time out at 7529.974237s 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.640+0000 7fc2ca7fc640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc2cc142a20 0x7fc2cc13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49700/0 (socket says 172.21.15.191:49700) 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.640+0000 7fc2cb7fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fc2cc0a96b0 0x7fc2cc13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.191:42052/0 (socket says 172.21.15.191:42052) 2024-03-25T18:14:07.641 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.640+0000 7fc2ca7fc640 1 -- 172.21.15.191:0/2428592382 learned_addr learned my addr 172.21.15.191:0/2428592382 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:07.643 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398647742 0 0) 0x7fc2cc13ff10 con 0x7fc2cc142a20 2024-03-25T18:14:07.643 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.643 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.643 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.643 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:14:07.643 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b4003520 con 0x7fc2cc142a20 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460776251 0 0) 0x7fc2cc0a9d60 con 0x7fc2cc0b4700 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.642+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc13ff10 con 0x7fc2cc0b4700 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.643+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602747595 0 0) 0x7fc2cc0aa1f0 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.643+0000 7fc2c9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.643+0000 7fc2c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.643+0000 7fc2c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.643+0000 7fc2c9ffb640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:14:07.644 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.643+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc0a9d60 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124624640 0 0) 0x7fc2b4003520 con 0x7fc2cc142a20 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 >> v1:172.21.15.162:6790/0 conn(0x7fc2cc0a96b0 legacy=0x7fc2cc13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 >> v1:172.21.15.162:6789/0 conn(0x7fc2cc0b4700 legacy=0x7fc2cc142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc0b4ae0 con 0x7fc2cc142a20 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.646 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc2bc001690 con 0x7fc2cc142a20 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2c9ffb640 10 monclient: dump: 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2d12a8640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:14:07.647 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.645+0000 7fc2d12a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:07.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2bc0017f0 con 0x7fc2cc142a20 2024-03-25T18:14:07.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2c9ffb640 1 -- 172.21.15.191:0/2428592382 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc2bc001c00 con 0x7fc2cc142a20 2024-03-25T18:14:07.673 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 4 set_mon_vals no callback set 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2c9ffb640 10 monclient: dump: 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.674 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 10 monclient: shutdown 2024-03-25T18:14:07.675 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2428592382 >> v1:172.21.15.191:6789/0 conn(0x7fc2cc142a20 legacy=0x7fc2cc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2428592382 shutdown_connections 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2428592382 >> 172.21.15.191:0/2428592382 conn(0x7fc2cc0668e0 msgr2=0x7fc2cc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2428592382 shutdown_connections 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.673+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2428592382 wait complete. 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.674+0000 7fc2d12a8640 10 monclient: build_initial_monmap 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.674+0000 7fc2d12a8640 10 monclient: monmap: 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:07.676 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.674+0000 7fc2d12a8640 1 librados: starting msgr at 2024-03-25T18:14:07.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.674+0000 7fc2d12a8640 1 librados: starting objecter 2024-03-25T18:14:07.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 1 Processor -- start 2024-03-25T18:14:07.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 1 -- start start 2024-03-25T18:14:07.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 1 librados: setting wanted keys 2024-03-25T18:14:07.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 1 librados: calling monclient init 2024-03-25T18:14:07.677 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 10 monclient: init 2024-03-25T18:14:07.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:07.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:07.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc2cc0a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:07.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc2cc0b4700 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:07.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc2cc142a20 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:07.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc0ae240 con 0x7fc2cc0b4700 2024-03-25T18:14:07.678 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc1d3fd0 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2cc1d4160 con 0x7fc2cc142a20 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.675+0000 7fc2d12a8640 10 monclient(hunting): authenticate will time out at 7530.010594s 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.676+0000 7fc2cb7fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fc2cc142a20 0x7fc2cc1daaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.191:42064/0 (socket says 172.21.15.191:42064) 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.676+0000 7fc2cb7fe640 1 -- 172.21.15.191:0/2787304831 learned_addr learned my addr 172.21.15.191:0/2787304831 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.676+0000 7fc2b37fe640 10 client.?.objecter ms_handle_connect 0x7fc2cc142a20 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.676+0000 7fc2b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.676+0000 7fc2b37fe640 10 client.?.objecter ms_handle_connect 0x7fc2cc0b4700 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.676+0000 7fc2b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.677+0000 7fc2b37fe640 10 client.?.objecter ms_handle_connect 0x7fc2cc0a96b0 2024-03-25T18:14:07.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.677+0000 7fc2b37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.678+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261807618 0 0) 0x7fc2cc0ae240 con 0x7fc2cc0b4700 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.678+0000 7fc2b37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.678+0000 7fc2b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.678+0000 7fc2b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.678+0000 7fc2b37fe640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.678+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b8004eb0 con 0x7fc2cc0b4700 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815436749 0 0) 0x7fc2cc1d4160 con 0x7fc2cc142a20 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:14:07.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc0ae240 con 0x7fc2cc142a20 2024-03-25T18:14:07.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368129617 0 0) 0x7fc2cc1d3fd0 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:07.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:07.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:07.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:14:07.681 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.679+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2cc1d4160 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2649969085 0 0) 0x7fc2b8004eb0 con 0x7fc2cc0b4700 2024-03-25T18:14:07.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2cc1d3fd0 con 0x7fc2cc0b4700 2024-03-25T18:14:07.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc2bc001b00 con 0x7fc2cc0b4700 2024-03-25T18:14:07.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.684 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 10 monclient(hunting): dump: 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3400013916 0 0) 0x7fc2cc0ae240 con 0x7fc2cc142a20 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2b8004eb0 con 0x7fc2cc142a20 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc2c4001be0 con 0x7fc2cc142a20 2024-03-25T18:14:07.685 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.683+0000 7fc2b37fe640 10 monclient(hunting): dump: 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.684+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1335285855 0 0) 0x7fc2cc1d4160 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.684+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2cc0ae240 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.684+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc2c00018f0 con 0x7fc2cc0a96b0 2024-03-25T18:14:07.686 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.684+0000 7fc2b37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.684+0000 7fc2b37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.684+0000 7fc2b37fe640 10 monclient(hunting): dump: 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.687 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.688 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 627695823 0 0) 0x7fc2b8004eb0 con 0x7fc2cc142a20 2024-03-25T18:14:07.688 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 >> v1:172.21.15.191:6789/0 conn(0x7fc2cc0a96b0 legacy=0x7fc2cc0ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.688 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 >> v1:172.21.15.162:6789/0 conn(0x7fc2cc0b4700 legacy=0x7fc2cc1d4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:07.688 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:07.688 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 1 monclient: found mon.c 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc0acee0 con 0x7fc2cc142a20 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 5 monclient: authenticate success, global_id 4130 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 10 monclient: _renew_subs 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2cc1d4d40 con 0x7fc2cc142a20 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 10 monclient: _renew_subs 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2cc1db3e0 con 0x7fc2cc142a20 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.686+0000 7fc2d12a8640 1 librados: init done 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.687+0000 7fc2d12a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.687+0000 7fc2d12a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.689 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.687+0000 7fc2d12a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:07.690 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.687+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2cc0ad130 con 0x7fc2cc142a20 2024-03-25T18:14:07.769 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:07.769+0000 7f214ce32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.879+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2c40020f0 con 0x7fc2cc142a20 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.879+0000 7fc2b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.879+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc2c4004d00 con 0x7fc2cc142a20 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.879+0000 7fc2b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.879+0000 7fc2b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.879+0000 7fc2b37fe640 10 monclient: dump: 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:07.880 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:07.881 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:07.881 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:07.881 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:07.881 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:07.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.879+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc2c4005230 con 0x7fc2cc142a20 2024-03-25T18:14:07.881 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.879+0000 7fc2c97fa640 4 set_mon_vals no callback set 2024-03-25T18:14:07.882 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.881+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2070401940 0 0) 0x7fc2c4006810 con 0x7fc2cc142a20 2024-03-25T18:14:07.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.881+0000 7fc2b37fe640 10 client.4130.objecter ms_dispatch 0x7fc2cc000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:14:07.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.881+0000 7fc2b37fe640 3 client.4130.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:14:07.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.881+0000 7fc2b37fe640 3 client.4130.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:14:07.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.881+0000 7fc2b37fe640 20 client.4130.objecter dump_active .. 0 homeless 2024-03-25T18:14:07.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.881+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc2c4006bc0 con 0x7fc2cc142a20 2024-03-25T18:14:07.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.882+0000 7fc2b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:07.883 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.882+0000 7fc2b37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:07.909 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:07.908+0000 7f214ce32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:14:07.932 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:07.931+0000 7f49ca04c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:14:07.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.954+0000 7fc2d12a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "24966f74-30d0-4d45-9c63-61d635d94d26", "id": 2}] 2024-03-25T18:14:07.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.954+0000 7fc2d12a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "24966f74-30d0-4d45-9c63-61d635d94d26", "id": 2}] 2024-03-25T18:14:07.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.954+0000 7fc2d12a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:07.955 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:07.954+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd new", "uuid": "24966f74-30d0-4d45-9c63-61d635d94d26", "id": 2} v 0) v1 -- 0x7fc2cc04f730 con 0x7fc2cc142a20 2024-03-25T18:14:08.062 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.061+0000 7fc2b37fe640 1 -- 172.21.15.191:0/2787304831 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "24966f74-30d0-4d45-9c63-61d635d94d26", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 668096579 0 216599125) 0x7fc2c40064c0 con 0x7fc2cc142a20 2024-03-25T18:14:08.063 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.061+0000 7fc2b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "24966f74-30d0-4d45-9c63-61d635d94d26", "id": 2}] 2024-03-25T18:14:08.063 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.061+0000 7fc2b37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:08.063 INFO:teuthology.orchestra.run.smithi191.stdout:2 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 10 librados: watch_flush enter 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 10 librados: watch_flush exit 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 20 client.4130.objecter shutdown clearing up homeless session... 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 10 client.4130.objecter successfully canceled tick 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 10 monclient: shutdown 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 >> v1:172.21.15.162:6790/0 conn(0x7fc2cc142a20 legacy=0x7fc2cc1daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 shutdown_connections 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 >> 172.21.15.191:0/2787304831 conn(0x7fc2cc0668e0 msgr2=0x7fc2cc0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 shutdown_connections 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 1 -- 172.21.15.191:0/2787304831 wait complete. 2024-03-25T18:14:08.064 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.062+0000 7fc2d12a8640 1 librados: shutdown 2024-03-25T18:14:08.071 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:08.070+0000 7f49ca04c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:14:08.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new cf0e6d69-1049-4141-bfd3-3d590e7936f6 3 2024-03-25T18:14:08.172 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.172+0000 7f5b35efb640 10 monclient: get_monmap_and_config 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.172+0000 7f5b35efb640 10 monclient: build_initial_monmap 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.172+0000 7f5b35efb640 10 monclient: monmap: 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:14:08.173228+0000 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:14:08.173228+0000 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:08.173 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.174 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.172+0000 7f5b35efb640 1 Processor -- start 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.172+0000 7f5b35efb640 1 -- start start 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.172+0000 7f5b35efb640 10 monclient: init 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 10 monclient(hunting): picked mon.noname-a con 0x7f5b300b4700 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 10 monclient(hunting): picked mon.noname-b con 0x7f5b30142a20 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 10 monclient(hunting): picked mon.noname-c con 0x7f5b300a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b300a9d60 con 0x7f5b300b4700 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3013ff10 con 0x7f5b300a96b0 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b300aa1f0 con 0x7f5b30142a20 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.173+0000 7f5b35efb640 10 monclient(hunting): authenticate will time out at 7530.508337s 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.174+0000 7f5b2f7fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f5b300b4700 0x7f5b30142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.191:50876/0 (socket says 172.21.15.191:50876) 2024-03-25T18:14:08.175 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.174+0000 7f5b2f7fe640 1 -- 172.21.15.191:0/3235208204 learned_addr learned my addr 172.21.15.191:0/3235208204 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:08.177 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171251459 0 0) 0x7f5b3013ff10 con 0x7f5b300a96b0 2024-03-25T18:14:08.177 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.177 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.177 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.177 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b18003520 con 0x7f5b300a96b0 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701188273 0 0) 0x7f5b300a9d60 con 0x7f5b300b4700 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b3013ff10 con 0x7f5b300b4700 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667155110 0 0) 0x7f5b300aa1f0 con 0x7f5b30142a20 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:14:08.178 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.176+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b300a9d60 con 0x7f5b30142a20 2024-03-25T18:14:08.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285068932 0 0) 0x7f5b18003520 con 0x7f5b300a96b0 2024-03-25T18:14:08.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 >> v1:172.21.15.162:6790/0 conn(0x7f5b30142a20 legacy=0x7f5b3013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 >> v1:172.21.15.162:6789/0 conn(0x7f5b300b4700 legacy=0x7f5b30142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:08.180 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b300b4ae0 con 0x7f5b300a96b0 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5b28001940 con 0x7f5b300a96b0 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b35efb640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b35efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b2e7fc640 10 monclient: dump: 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.181 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.179+0000 7f5b35efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.181+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b28001aa0 con 0x7f5b300a96b0 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.181+0000 7f5b2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:08.182 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.181+0000 7f5b2e7fc640 1 -- 172.21.15.191:0/3235208204 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5b28001eb0 con 0x7f5b300a96b0 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.181+0000 7f5b35efb640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.181+0000 7f5b35efb640 4 set_mon_vals no callback set 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.181+0000 7f5b2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.181+0000 7f5b2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.181+0000 7f5b2e7fc640 10 monclient: dump: 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.183 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.182+0000 7f5b35efb640 10 monclient: shutdown 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.182+0000 7f5b35efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.182+0000 7f5b35efb640 1 -- 172.21.15.191:0/3235208204 >> v1:172.21.15.191:6789/0 conn(0x7f5b300a96b0 legacy=0x7f5b3013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.182+0000 7f5b35efb640 1 -- 172.21.15.191:0/3235208204 shutdown_connections 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.182+0000 7f5b35efb640 1 -- 172.21.15.191:0/3235208204 >> 172.21.15.191:0/3235208204 conn(0x7f5b300668e0 msgr2=0x7f5b30066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.182+0000 7f5b35efb640 1 -- 172.21.15.191:0/3235208204 shutdown_connections 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.182+0000 7f5b35efb640 1 -- 172.21.15.191:0/3235208204 wait complete. 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 10 monclient: build_initial_monmap 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 10 monclient: monmap: 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-25T18:14:08.186 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 1 librados: starting msgr at 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 1 librados: starting objecter 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 1 Processor -- start 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 1 -- start start 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 1 librados: setting wanted keys 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 1 librados: calling monclient init 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.183+0000 7f5b35efb640 10 monclient: init 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 10 monclient(hunting): picked mon.noname-a con 0x7f5b300a96b0 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 10 monclient(hunting): picked mon.noname-b con 0x7f5b300b4700 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 10 monclient(hunting): picked mon.noname-c con 0x7f5b30142a20 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:08.187 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b30138cc0 con 0x7f5b300a96b0 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b30139150 con 0x7f5b300b4700 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b30131970 con 0x7f5b30142a20 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b35efb640 10 monclient(hunting): authenticate will time out at 7530.519264s 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b2effd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f5b300b4700 0x7f5b30130b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49724/0 (socket says 172.21.15.191:49724) 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.184+0000 7f5b2effd640 1 -- 172.21.15.191:0/2854034756 learned_addr learned my addr 172.21.15.191:0/2854034756 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.185+0000 7f5b17fff640 10 client.?.objecter ms_handle_connect 0x7f5b300b4700 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.185+0000 7f5b17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.185+0000 7f5b17fff640 10 client.?.objecter ms_handle_connect 0x7f5b300a96b0 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.185+0000 7f5b17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.185+0000 7f5b17fff640 10 client.?.objecter ms_handle_connect 0x7f5b30142a20 2024-03-25T18:14:08.188 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.185+0000 7f5b17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.189 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337251892 0 0) 0x7f5b30139150 con 0x7f5b300b4700 2024-03-25T18:14:08.189 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.189 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.189 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.189 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b1c004eb0 con 0x7f5b300b4700 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559291393 0 0) 0x7f5b30131970 con 0x7f5b30142a20 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30139150 con 0x7f5b30142a20 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288188945 0 0) 0x7f5b30138cc0 con 0x7f5b300a96b0 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:14:08.190 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.187+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b30131970 con 0x7f5b300a96b0 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 443355240 0 0) 0x7f5b30139150 con 0x7f5b30142a20 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b30138cc0 con 0x7f5b30142a20 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3258377327 0 0) 0x7f5b1c004eb0 con 0x7f5b300b4700 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b30139150 con 0x7f5b300b4700 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5b24001970 con 0x7f5b300b4700 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 10 monclient(hunting): dump: 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.193 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.191+0000 7f5b17fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.192+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1812176227 0 0) 0x7f5b30131970 con 0x7f5b300a96b0 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.192+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b1c004eb0 con 0x7f5b300a96b0 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.192+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5b20001930 con 0x7f5b300a96b0 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.192+0000 7f5b17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.192+0000 7f5b17fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.192+0000 7f5b17fff640 10 monclient(hunting): dump: 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.194 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.195 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.195 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.195 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.195 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.195 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.195 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.195 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:08.193+0000 7f49ca04c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2683357913 0 0) 0x7f5b1c004eb0 con 0x7f5b300a96b0 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 >> v1:172.21.15.162:6790/0 conn(0x7f5b30142a20 legacy=0x7f5b30131260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 >> v1:172.21.15.191:6789/0 conn(0x7f5b300b4700 legacy=0x7f5b30130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 1 monclient: found mon.a 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3013a4b0 con 0x7f5b300a96b0 2024-03-25T18:14:08.196 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 10 monclient: _renew_subs 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b301387a0 con 0x7f5b300a96b0 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 10 monclient: _renew_subs 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b30139510 con 0x7f5b300a96b0 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.195+0000 7f5b35efb640 1 librados: init done 2024-03-25T18:14:08.197 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.196+0000 7f5b35efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:08.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.196+0000 7f5b35efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:08.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.196+0000 7f5b35efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:08.198 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.196+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b300b3df0 con 0x7f5b300a96b0 2024-03-25T18:14:08.240 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:08.239+0000 7f49ca04c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:14:08.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b200047e0 con 0x7f5b300a96b0 2024-03-25T18:14:08.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:08.292 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5b20001d80 con 0x7f5b300a96b0 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 10 monclient: dump: 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5b20004f40 con 0x7f5b300a96b0 2024-03-25T18:14:08.293 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b2dffb640 4 set_mon_vals no callback set 2024-03-25T18:14:08.294 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2492830555 0 0) 0x7f5b20006650 con 0x7f5b300a96b0 2024-03-25T18:14:08.294 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 10 client.4137.objecter ms_dispatch 0x7f5b30000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:14:08.294 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:14:08.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:14:08.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T18:14:08.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5b200061c0 con 0x7f5b300a96b0 2024-03-25T18:14:08.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:08.295 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.292+0000 7f5b17fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:08.329 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:08.329+0000 7f214ce32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:14:08.353 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:08.352+0000 7f49ca04c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.365+0000 7f5b35efb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "cf0e6d69-1049-4141-bfd3-3d590e7936f6", "id": 3}] 2024-03-25T18:14:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.365+0000 7f5b35efb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "cf0e6d69-1049-4141-bfd3-3d590e7936f6", "id": 3}] 2024-03-25T18:14:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.365+0000 7f5b35efb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:08.366 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.365+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd new", "uuid": "cf0e6d69-1049-4141-bfd3-3d590e7936f6", "id": 3} v 0) v1 -- 0x7f5b3004f730 con 0x7f5b300a96b0 2024-03-25T18:14:08.375 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:08.374+0000 7f214ce32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:14:08.420 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.420+0000 7f5b17fff640 1 -- 172.21.15.191:0/2854034756 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cf0e6d69-1049-4141-bfd3-3d590e7936f6", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2301850668 0 525046306) 0x7f5b200064b0 con 0x7f5b300a96b0 2024-03-25T18:14:08.421 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.420+0000 7f5b17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "cf0e6d69-1049-4141-bfd3-3d590e7936f6", "id": 3}] 2024-03-25T18:14:08.421 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.420+0000 7f5b17fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:08.421 INFO:teuthology.orchestra.run.smithi191.stdout:3 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.420+0000 7f5b35efb640 10 librados: watch_flush enter 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 10 librados: watch_flush exit 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 10 client.4137.objecter successfully canceled tick 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 10 monclient: shutdown 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 >> v1:172.21.15.162:6789/0 conn(0x7f5b300a96b0 legacy=0x7f5b301d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 shutdown_connections 2024-03-25T18:14:08.422 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 >> 172.21.15.191:0/2854034756 conn(0x7f5b300668e0 msgr2=0x7f5b300b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:08.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 shutdown_connections 2024-03-25T18:14:08.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 1 -- 172.21.15.191:0/2854034756 wait complete. 2024-03-25T18:14:08.423 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.421+0000 7f5b35efb640 1 librados: shutdown 2024-03-25T18:14:08.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 9e7e1c42-7fbc-4ced-a70b-4556c30fbc66 4 2024-03-25T18:14:08.455 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:08.454+0000 7f49ca04c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:14:08.498 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:08.497+0000 7f214ce32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.531+0000 7fdf7056d640 10 monclient: get_monmap_and_config 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.531+0000 7fdf7056d640 10 monclient: build_initial_monmap 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.531+0000 7fdf7056d640 10 monclient: monmap: 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:14:08.532687+0000 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:14:08.532687+0000 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:08.532 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.533 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:08.533 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:08.533 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:08.533 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.534 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.532+0000 7fdf7056d640 1 Processor -- start 2024-03-25T18:14:08.534 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.532+0000 7fdf7056d640 1 -- start start 2024-03-25T18:14:08.534 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.532+0000 7fdf7056d640 10 monclient: init 2024-03-25T18:14:08.534 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.532+0000 7fdf7056d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:08.534 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.532+0000 7fdf7056d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:08.534 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.532+0000 7fdf7056d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf680b4700 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:08.534 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf7056d640 10 monclient(hunting): picked mon.noname-b con 0x7fdf68142a20 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:08.534 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf7056d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf680a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf7056d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf680a9d60 con 0x7fdf680b4700 2024-03-25T18:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf7056d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6813ff10 con 0x7fdf680a96b0 2024-03-25T18:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf7056d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf680aa1f0 con 0x7fdf68142a20 2024-03-25T18:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf7056d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf7056d640 10 monclient(hunting): authenticate will time out at 7530.867845s 2024-03-25T18:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf6eb0a640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fdf680a96b0 0x7fdf6813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49732/0 (socket says 172.21.15.191:49732) 2024-03-25T18:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf6eb0a640 1 -- 172.21.15.191:0/944289483 learned_addr learned my addr 172.21.15.191:0/944289483 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.533+0000 7fdf6e309640 1 --1- 172.21.15.191:0/944289483 >> v1:172.21.15.162:6789/0 conn(0x7fdf680b4700 0x7fdf68142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.191:50896/0 (socket says 172.21.15.191:50896) 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.535+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168513690 0 0) 0x7fdf6813ff10 con 0x7fdf680a96b0 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.535+0000 7fdf6d307640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.535+0000 7fdf6d307640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.535+0000 7fdf6d307640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.535+0000 7fdf6d307640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.535+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf54003520 con 0x7fdf680a96b0 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193901583 0 0) 0x7fdf680a9d60 con 0x7fdf680b4700 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6813ff10 con 0x7fdf680b4700 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774844862 0 0) 0x7fdf680aa1f0 con 0x7fdf68142a20 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.537 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.538 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.538 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:14:08.538 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.536+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf680a9d60 con 0x7fdf68142a20 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2878107313 0 0) 0x7fdf54003520 con 0x7fdf680a96b0 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 >> v1:172.21.15.162:6790/0 conn(0x7fdf68142a20 legacy=0x7fdf6813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 >> v1:172.21.15.162:6789/0 conn(0x7fdf680b4700 legacy=0x7fdf68142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 10 monclient: _finish_hunting 0 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 1 monclient: found mon.noname-c 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680b4ae0 con 0x7fdf680a96b0 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 10 monclient: _finish_auth 0 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 10 monclient: _check_auth_tickets 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdf60001a10 con 0x7fdf680a96b0 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf7056d640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf7056d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:08.540 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf6d307640 10 monclient: dump: 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.541 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.539+0000 7fdf7056d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.540+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf60001b70 con 0x7fdf680a96b0 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.540+0000 7fdf6d307640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.540+0000 7fdf6d307640 1 -- 172.21.15.191:0/944289483 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdf60001f80 con 0x7fdf680a96b0 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.540+0000 7fdf7056d640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.540+0000 7fdf7056d640 4 set_mon_vals no callback set 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.540+0000 7fdf6d307640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.540+0000 7fdf6d307640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.540+0000 7fdf6d307640 10 monclient: dump: 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.543 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.541+0000 7fdf7056d640 10 monclient: shutdown 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.541+0000 7fdf7056d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.541+0000 7fdf7056d640 1 -- 172.21.15.191:0/944289483 >> v1:172.21.15.191:6789/0 conn(0x7fdf680a96b0 legacy=0x7fdf6813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.541+0000 7fdf7056d640 1 -- 172.21.15.191:0/944289483 shutdown_connections 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.541+0000 7fdf7056d640 1 -- 172.21.15.191:0/944289483 >> 172.21.15.191:0/944289483 conn(0x7fdf680668e0 msgr2=0x7fdf68066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.541+0000 7fdf7056d640 1 -- 172.21.15.191:0/944289483 shutdown_connections 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.541+0000 7fdf7056d640 1 -- 172.21.15.191:0/944289483 wait complete. 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 10 monclient: build_initial_monmap 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 10 monclient: monmap: 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:08.544 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 1 librados: starting msgr at 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 1 librados: starting objecter 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 1 Processor -- start 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 1 -- start start 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 1 librados: setting wanted keys 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 1 librados: calling monclient init 2024-03-25T18:14:08.545 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.542+0000 7fdf7056d640 10 monclient: init 2024-03-25T18:14:08.547 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:08.547 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:08.547 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 10 monclient(hunting): picked mon.noname-b con 0x7fdf680a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:08.547 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 10 monclient(hunting): picked mon.noname-a con 0x7fdf680b4700 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 10 monclient(hunting): picked mon.noname-c con 0x7fdf68142a20 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6813b3e0 con 0x7fdf680b4700 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf680ae520 con 0x7fdf680a96b0 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf680ab880 con 0x7fdf68142a20 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf7056d640 10 monclient(hunting): authenticate will time out at 7530.877924s 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf6eb0a640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fdf68142a20 0x7fdf680b0c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.191:42098/0 (socket says 172.21.15.191:42098) 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.543+0000 7fdf6eb0a640 1 -- 172.21.15.191:0/3154915138 learned_addr learned my addr 172.21.15.191:0/3154915138 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.544+0000 7fdf5effd640 10 client.?.objecter ms_handle_connect 0x7fdf68142a20 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.544+0000 7fdf5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.544+0000 7fdf5effd640 10 client.?.objecter ms_handle_connect 0x7fdf680b4700 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.544+0000 7fdf5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.545+0000 7fdf5effd640 10 client.?.objecter ms_handle_connect 0x7fdf680a96b0 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.545+0000 7fdf5effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756756910 0 0) 0x7fdf6813b3e0 con 0x7fdf680b4700 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.548 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c004eb0 con 0x7fdf680b4700 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393649544 0 0) 0x7fdf680ab880 con 0x7fdf68142a20 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.546+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6813b3e0 con 0x7fdf68142a20 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.547+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919356713 0 0) 0x7fdf680ae520 con 0x7fdf680a96b0 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.547+0000 7fdf5effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.547+0000 7fdf5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.547+0000 7fdf5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.547+0000 7fdf5effd640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:14:08.549 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.547+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf680ab880 con 0x7fdf680a96b0 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2698731195 0 0) 0x7fdf4c004eb0 con 0x7fdf680b4700 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf680ae520 con 0x7fdf680b4700 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdf580019b0 con 0x7fdf680b4700 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 10 monclient(hunting): dump: 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.552 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1897400861 0 0) 0x7fdf680ab880 con 0x7fdf680a96b0 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf4c004eb0 con 0x7fdf680a96b0 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdf640018f0 con 0x7fdf680a96b0 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.551+0000 7fdf5effd640 10 monclient(hunting): dump: 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.553 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.554 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.554 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.554 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.554 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.554 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.554 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.554 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.553+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1384113251 0 0) 0x7fdf6813b3e0 con 0x7fdf68142a20 2024-03-25T18:14:08.554 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.553+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf680ab880 con 0x7fdf68142a20 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.553+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdf60001e80 con 0x7fdf68142a20 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.553+0000 7fdf5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.553+0000 7fdf5effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.553+0000 7fdf5effd640 10 monclient(hunting): dump: 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.555 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3703926764 0 0) 0x7fdf4c004eb0 con 0x7fdf680a96b0 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 >> v1:172.21.15.162:6790/0 conn(0x7fdf68142a20 legacy=0x7fdf680b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 >> v1:172.21.15.162:6789/0 conn(0x7fdf680b4700 legacy=0x7fdf68139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 1 monclient: found mon.b 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680aba10 con 0x7fdf680a96b0 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 10 monclient: _finish_auth 0 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:14:08.557 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 10 monclient: _renew_subs 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6813a490 con 0x7fdf680a96b0 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 10 monclient: _renew_subs 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf680ad430 con 0x7fdf680a96b0 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.555+0000 7fdf7056d640 1 librados: init done 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.556+0000 7fdf7056d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.556+0000 7fdf7056d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.556+0000 7fdf7056d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:08.558 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.556+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6804f730 con 0x7fdf680a96b0 2024-03-25T18:14:08.602 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:08.601+0000 7f214ce32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:14:08.621 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:08.620+0000 7f49ca04c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:08.664 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:08.664+0000 7f49ca04c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf640047e0 con 0x7fdf680a96b0 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdf64001d80 con 0x7fdf680a96b0 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 10 monclient: dump: 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:08.749 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdf64004f00 con 0x7fdf680a96b0 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf6cb06640 4 set_mon_vals no callback set 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2544176382 0 0) 0x7fdf640065b0 con 0x7fdf680a96b0 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 10 client.4138.objecter ms_dispatch 0x7fdf68000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdf640069d0 con 0x7fdf680a96b0 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:08.750 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.748+0000 7fdf5effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:08.757 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:08.756+0000 7f214ce32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:08.800 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:08.799+0000 7f214ce32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:08.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.821+0000 7fdf7056d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9e7e1c42-7fbc-4ced-a70b-4556c30fbc66", "id": 4}] 2024-03-25T18:14:08.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.821+0000 7fdf7056d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9e7e1c42-7fbc-4ced-a70b-4556c30fbc66", "id": 4}] 2024-03-25T18:14:08.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.821+0000 7fdf7056d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:08.822 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.821+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9e7e1c42-7fbc-4ced-a70b-4556c30fbc66", "id": 4} v 0) v1 -- 0x7fdf68142220 con 0x7fdf680a96b0 2024-03-25T18:14:08.823 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:08.822+0000 7f49ca04c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:14:08.941 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.940+0000 7fdf5effd640 1 -- 172.21.15.191:0/3154915138 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e7e1c42-7fbc-4ced-a70b-4556c30fbc66", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2764516126 0 1697012583) 0x7fdf64006170 con 0x7fdf680a96b0 2024-03-25T18:14:08.941 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.940+0000 7fdf5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9e7e1c42-7fbc-4ced-a70b-4556c30fbc66", "id": 4}] 2024-03-25T18:14:08.941 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.940+0000 7fdf5effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:08.941 INFO:teuthology.orchestra.run.smithi191.stdout:4 2024-03-25T18:14:08.942 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 10 librados: watch_flush enter 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 10 librados: watch_flush exit 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 10 client.4138.objecter successfully canceled tick 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 10 monclient: shutdown 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 >> v1:172.21.15.191:6789/0 conn(0x7fdf680a96b0 legacy=0x7fdf680ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 shutdown_connections 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 >> 172.21.15.191:0/3154915138 conn(0x7fdf680668e0 msgr2=0x7fdf6813c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 shutdown_connections 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 1 -- 172.21.15.191:0/3154915138 wait complete. 2024-03-25T18:14:08.943 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:08.941+0000 7fdf7056d640 1 librados: shutdown 2024-03-25T18:14:08.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 00c78472-8c62-45cc-bcd8-1ac19b38de5e 5 2024-03-25T18:14:08.961 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:08.960+0000 7f214ce32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.091+0000 7fb2cacb3640 10 monclient: get_monmap_and_config 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.091+0000 7fb2cacb3640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.091+0000 7fb2cacb3640 10 monclient: monmap: 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:14:09.092988+0000 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:14:09.092988+0000 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.093 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:09.094 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:09.094 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:09.094 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.094 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.092+0000 7fb2cacb3640 1 Processor -- start 2024-03-25T18:14:09.094 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.092+0000 7fb2cacb3640 1 -- start start 2024-03-25T18:14:09.094 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.092+0000 7fb2cacb3640 10 monclient: init 2024-03-25T18:14:09.095 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.095 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:09.095 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c40b4700 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:09.095 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c4142a20 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:09.095 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c40a96b0 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:09.095 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40a9d60 con 0x7fb2c40b4700 2024-03-25T18:14:09.095 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c413ff10 con 0x7fb2c4142a20 2024-03-25T18:14:09.096 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40aa1f0 con 0x7fb2c40a96b0 2024-03-25T18:14:09.096 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.096 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2cacb3640 10 monclient(hunting): authenticate will time out at 7531.428124s 2024-03-25T18:14:09.096 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2c3fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb2c4142a20 0x7fb2c413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49752/0 (socket says 172.21.15.191:49752) 2024-03-25T18:14:09.096 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.093+0000 7fb2c3fff640 1 -- 172.21.15.191:0/4272316955 learned_addr learned my addr 172.21.15.191:0/4272316955 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:09.097 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568943709 0 0) 0x7fb2c40a9d60 con 0x7fb2c40b4700 2024-03-25T18:14:09.097 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.097 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b4003520 con 0x7fb2c40b4700 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403058852 0 0) 0x7fb2c40aa1f0 con 0x7fb2c40a96b0 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c40a9d60 con 0x7fb2c40a96b0 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 616355966 0 0) 0x7fb2c413ff10 con 0x7fb2c4142a20 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:14:09.099 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.096+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c40aa1f0 con 0x7fb2c4142a20 2024-03-25T18:14:09.100 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 546945413 0 0) 0x7fb2c40a9d60 con 0x7fb2c40a96b0 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 >> v1:172.21.15.191:6789/0 conn(0x7fb2c4142a20 legacy=0x7fb2c413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 >> v1:172.21.15.162:6789/0 conn(0x7fb2c40b4700 legacy=0x7fb2c4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 1 monclient: found mon.noname-b 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40b4ae0 con 0x7fb2c40a96b0 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2bc0019f0 con 0x7fb2c40a96b0 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2cacb3640 5 monclient: authenticate success, global_id 4145 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2cacb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2c37fe640 10 monclient: dump: 2024-03-25T18:14:09.101 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.102 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.099+0000 7fb2cacb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2bc001b50 con 0x7fb2c40a96b0 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2cacb3640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2cacb3640 4 set_mon_vals no callback set 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2c37fe640 1 -- 172.21.15.191:0/4272316955 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2bc001fa0 con 0x7fb2c40a96b0 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2c37fe640 10 monclient: dump: 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.105 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2cacb3640 10 monclient: shutdown 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2cacb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2cacb3640 1 -- 172.21.15.191:0/4272316955 >> v1:172.21.15.162:6790/0 conn(0x7fb2c40a96b0 legacy=0x7fb2c413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2cacb3640 1 -- 172.21.15.191:0/4272316955 shutdown_connections 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.101+0000 7fb2cacb3640 1 -- 172.21.15.191:0/4272316955 >> 172.21.15.191:0/4272316955 conn(0x7fb2c40668e0 msgr2=0x7fb2c4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.102+0000 7fb2cacb3640 1 -- 172.21.15.191:0/4272316955 shutdown_connections 2024-03-25T18:14:09.106 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.102+0000 7fb2cacb3640 1 -- 172.21.15.191:0/4272316955 wait complete. 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.102+0000 7fb2cacb3640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.102+0000 7fb2cacb3640 10 monclient: monmap: 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.102+0000 7fb2cacb3640 1 librados: starting msgr at 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.102+0000 7fb2cacb3640 1 librados: starting objecter 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 1 Processor -- start 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 1 -- start start 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 1 librados: setting wanted keys 2024-03-25T18:14:09.107 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 1 librados: calling monclient init 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 10 monclient: init 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c40a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c40b4700 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c4142a20 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c4135b60 con 0x7fb2c4142a20 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c41349f0 con 0x7fb2c40a96b0 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c4134b80 con 0x7fb2c40b4700 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.103+0000 7fb2cacb3640 10 monclient(hunting): authenticate will time out at 7531.438582s 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.104+0000 7fb2c3fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fb2c40b4700 0x7fb2c41d7b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.191:42120/0 (socket says 172.21.15.191:42120) 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.104+0000 7fb2c3fff640 1 -- 172.21.15.191:0/2300629378 learned_addr learned my addr 172.21.15.191:0/2300629378 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.104+0000 7fb2c17fa640 10 client.?.objecter ms_handle_connect 0x7fb2c40a96b0 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.104+0000 7fb2c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.104+0000 7fb2c17fa640 10 client.?.objecter ms_handle_connect 0x7fb2c4142a20 2024-03-25T18:14:09.108 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.104+0000 7fb2c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.109 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.104+0000 7fb2c17fa640 10 client.?.objecter ms_handle_connect 0x7fb2c40b4700 2024-03-25T18:14:09.109 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.104+0000 7fb2c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.109 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293932003 0 0) 0x7fb2c4134b80 con 0x7fb2c40b4700 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2a8004eb0 con 0x7fb2c40b4700 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315807628 0 0) 0x7fb2c4135b60 con 0x7fb2c4142a20 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.106+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4134b80 con 0x7fb2c4142a20 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.107+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927534963 0 0) 0x7fb2c41349f0 con 0x7fb2c40a96b0 2024-03-25T18:14:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.107+0000 7fb2c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.107+0000 7fb2c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.107+0000 7fb2c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.107+0000 7fb2c17fa640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:14:09.111 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.107+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c4135b60 con 0x7fb2c40a96b0 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 459409549 0 0) 0x7fb2a8004eb0 con 0x7fb2c40b4700 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c41349f0 con 0x7fb2c40b4700 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2b0001870 con 0x7fb2c40b4700 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 10 monclient(hunting): dump: 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.112 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1867769826 0 0) 0x7fb2c4134b80 con 0x7fb2c4142a20 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2a8004eb0 con 0x7fb2c4142a20 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2bc001e60 con 0x7fb2c4142a20 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.111+0000 7fb2c17fa640 10 monclient(hunting): dump: 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.113 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.114 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.114 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.114 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.114 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.114 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.114 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.114 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.115 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.113+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2555107765 0 0) 0x7fb2c4135b60 con 0x7fb2c40a96b0 2024-03-25T18:14:09.115 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.113+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c4134b80 con 0x7fb2c40a96b0 2024-03-25T18:14:09.115 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.113+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2b8001230 con 0x7fb2c40a96b0 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.113+0000 7fb2c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.113+0000 7fb2c17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.113+0000 7fb2c17fa640 10 monclient(hunting): dump: 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3968042172 0 0) 0x7fb2a8004eb0 con 0x7fb2c4142a20 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 >> v1:172.21.15.162:6790/0 conn(0x7fb2c40b4700 legacy=0x7fb2c41d7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 >> v1:172.21.15.191:6789/0 conn(0x7fb2c40a96b0 legacy=0x7fb2c41d7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.116 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 1 monclient: found mon.a 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c4134d10 con 0x7fb2c4142a20 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 5 monclient: authenticate success, global_id 4149 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 10 monclient: _renew_subs 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c41416c0 con 0x7fb2c4142a20 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 10 monclient: _renew_subs 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c4135670 con 0x7fb2c4142a20 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.114+0000 7fb2cacb3640 1 librados: init done 2024-03-25T18:14:09.117 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.115+0000 7fb2cacb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.118 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.115+0000 7fb2cacb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.118 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.115+0000 7fb2cacb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.118 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.115+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c4135970 con 0x7fb2c4142a20 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2bc001590 con 0x7fb2c4142a20 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2bc003c20 con 0x7fb2c4142a20 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 10 monclient: dump: 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.217 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb2bc004190 con 0x7fb2c4142a20 2024-03-25T18:14:09.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c2ffd640 4 set_mon_vals no callback set 2024-03-25T18:14:09.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2819475243 0 0) 0x7fb2bc005870 con 0x7fb2c4142a20 2024-03-25T18:14:09.218 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 10 client.4149.objecter ms_dispatch 0x7fb2c4000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:14:09.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:14:09.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.216+0000 7fb2c17fa640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:14:09.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.217+0000 7fb2c17fa640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T18:14:09.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.217+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb2bc005be0 con 0x7fb2c4142a20 2024-03-25T18:14:09.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.217+0000 7fb2c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.219 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.217+0000 7fb2c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:09.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.289+0000 7fb2cacb3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "00c78472-8c62-45cc-bcd8-1ac19b38de5e", "id": 5}] 2024-03-25T18:14:09.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.289+0000 7fb2cacb3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "00c78472-8c62-45cc-bcd8-1ac19b38de5e", "id": 5}] 2024-03-25T18:14:09.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.289+0000 7fb2cacb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.290 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.289+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd new", "uuid": "00c78472-8c62-45cc-bcd8-1ac19b38de5e", "id": 5} v 0) v1 -- 0x7fb2c404f730 con 0x7fb2c4142a20 2024-03-25T18:14:09.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.318+0000 7fb2c17fa640 1 -- 172.21.15.191:0/2300629378 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00c78472-8c62-45cc-bcd8-1ac19b38de5e", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1093754353 0 1988412176) 0x7fb2bc0053c0 con 0x7fb2c4142a20 2024-03-25T18:14:09.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.318+0000 7fb2c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "00c78472-8c62-45cc-bcd8-1ac19b38de5e", "id": 5}] 2024-03-25T18:14:09.319 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.318+0000 7fb2c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:09.319 INFO:teuthology.orchestra.run.smithi191.stdout:5 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 10 librados: watch_flush enter 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 10 librados: watch_flush exit 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 10 client.4149.objecter successfully canceled tick 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 10 monclient: shutdown 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 >> v1:172.21.15.162:6789/0 conn(0x7fb2c4142a20 legacy=0x7fb2c41e1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 shutdown_connections 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.319+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 >> 172.21.15.191:0/2300629378 conn(0x7fb2c40668e0 msgr2=0x7fb2c40b6770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.320+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 shutdown_connections 2024-03-25T18:14:09.321 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.320+0000 7fb2cacb3640 1 -- 172.21.15.191:0/2300629378 wait complete. 2024-03-25T18:14:09.322 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.320+0000 7fb2cacb3640 1 librados: shutdown 2024-03-25T18:14:09.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new aad1d03e-a813-46d7-9e95-a932c7162531 6 2024-03-25T18:14:09.435 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.434+0000 7ff86428d640 10 monclient: get_monmap_and_config 2024-03-25T18:14:09.435 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.434+0000 7ff86428d640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.435 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.434+0000 7ff86428d640 10 monclient: monmap: 2024-03-25T18:14:09.435 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:14:09.435559+0000 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:14:09.435559+0000 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 1 Processor -- start 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 1 -- start start 2024-03-25T18:14:09.436 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 10 monclient: init 2024-03-25T18:14:09.437 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.437 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:09.437 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 10 monclient(hunting): picked mon.noname-b con 0x7ff85c0b4700 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:09.437 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c142a20 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:09.437 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 10 monclient(hunting): picked mon.noname-c con 0x7ff85c0a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0a9d60 con 0x7ff85c142a20 2024-03-25T18:14:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c13ff10 con 0x7ff85c0a96b0 2024-03-25T18:14:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0aa1f0 con 0x7ff85c0b4700 2024-03-25T18:14:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.435+0000 7ff86428d640 10 monclient(hunting): authenticate will time out at 7531.770755s 2024-03-25T18:14:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.436+0000 7ff86282a640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff85c0a96b0 0x7ff85c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49766/0 (socket says 172.21.15.191:49766) 2024-03-25T18:14:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.436+0000 7ff86282a640 1 -- 172.21.15.191:0/537802163 learned_addr learned my addr 172.21.15.191:0/537802163 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:09.439 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941907292 0 0) 0x7ff85c0a9d60 con 0x7ff85c142a20 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff848003520 con 0x7ff85c142a20 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705787026 0 0) 0x7ff85c13ff10 con 0x7ff85c0a96b0 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:14:09.440 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff85c0a9d60 con 0x7ff85c0a96b0 2024-03-25T18:14:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750797574 0 0) 0x7ff85c0aa1f0 con 0x7ff85c0b4700 2024-03-25T18:14:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:14:09.441 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.439+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff85c13ff10 con 0x7ff85c0b4700 2024-03-25T18:14:09.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4137495201 0 0) 0x7ff848003520 con 0x7ff85c142a20 2024-03-25T18:14:09.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 >> v1:172.21.15.162:6790/0 conn(0x7ff85c0b4700 legacy=0x7ff85c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 >> v1:172.21.15.191:6789/0 conn(0x7ff85c0a96b0 legacy=0x7ff85c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.443 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 1 monclient: found mon.noname-a 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c0b4ae0 con 0x7ff85c142a20 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff84c001690 con 0x7ff85c142a20 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff86428d640 5 monclient: authenticate success, global_id 4152 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff86428d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff861027640 10 monclient: dump: 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.444 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.445 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.445 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.442+0000 7ff86428d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.447 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff84c0017f0 con 0x7ff85c142a20 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff861027640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff861027640 1 -- 172.21.15.191:0/537802163 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff84c001c00 con 0x7ff85c142a20 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff86428d640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff86428d640 4 set_mon_vals no callback set 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff861027640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff861027640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff861027640 10 monclient: dump: 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.448 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.450 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff86428d640 10 monclient: shutdown 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff86428d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.447+0000 7ff86428d640 1 -- 172.21.15.191:0/537802163 >> v1:172.21.15.162:6789/0 conn(0x7ff85c142a20 legacy=0x7ff85c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.448+0000 7ff86428d640 1 -- 172.21.15.191:0/537802163 shutdown_connections 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.448+0000 7ff86428d640 1 -- 172.21.15.191:0/537802163 >> 172.21.15.191:0/537802163 conn(0x7ff85c0668e0 msgr2=0x7ff85c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.448+0000 7ff86428d640 1 -- 172.21.15.191:0/537802163 shutdown_connections 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.448+0000 7ff86428d640 1 -- 172.21.15.191:0/537802163 wait complete. 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.448+0000 7ff86428d640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.448+0000 7ff86428d640 10 monclient: monmap: 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:09.451 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.448+0000 7ff86428d640 1 librados: starting msgr at 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.448+0000 7ff86428d640 1 librados: starting objecter 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 1 Processor -- start 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 1 -- start start 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 1 librados: setting wanted keys 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 1 librados: calling monclient init 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 10 monclient: init 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 10 monclient(hunting): picked mon.noname-b con 0x7ff85c0a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 10 monclient(hunting): picked mon.noname-c con 0x7ff85c0b4700 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 10 monclient(hunting): picked mon.noname-a con 0x7ff85c142a20 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c13c170 con 0x7ff85c142a20 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c0abf30 con 0x7ff85c0a96b0 2024-03-25T18:14:09.452 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85c13c600 con 0x7ff85c0b4700 2024-03-25T18:14:09.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.449+0000 7ff86428d640 10 monclient(hunting): authenticate will time out at 7531.784423s 2024-03-25T18:14:09.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.450+0000 7ff862029640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ff85c0a96b0 0x7ff85c1387c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49782/0 (socket says 172.21.15.191:49782) 2024-03-25T18:14:09.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.450+0000 7ff862029640 1 -- 172.21.15.191:0/2411692634 learned_addr learned my addr 172.21.15.191:0/2411692634 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:09.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.450+0000 7ff852ffd640 10 client.?.objecter ms_handle_connect 0x7ff85c142a20 2024-03-25T18:14:09.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.450+0000 7ff852ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.453 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.452+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424302901 0 0) 0x7ff85c13c170 con 0x7ff85c142a20 2024-03-25T18:14:09.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.452+0000 7ff852ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.452+0000 7ff852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.452+0000 7ff852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.452+0000 7ff852ffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:14:09.454 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.452+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff840004eb0 con 0x7ff85c142a20 2024-03-25T18:14:09.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.457+0000 7ff852ffd640 10 client.?.objecter ms_handle_connect 0x7ff85c0b4700 2024-03-25T18:14:09.458 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.457+0000 7ff852ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2233040650 0 0) 0x7ff840004eb0 con 0x7ff85c142a20 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff85c13c170 con 0x7ff85c142a20 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 10 client.?.objecter ms_handle_connect 0x7ff85c0a96b0 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff854001bd0 con 0x7ff85c142a20 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.459 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.460 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.458+0000 7ff852ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2527436790 0 0) 0x7ff85c13c170 con 0x7ff85c142a20 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 >> v1:172.21.15.162:6790/0 conn(0x7ff85c0b4700 legacy=0x7ff85c131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 >> v1:172.21.15.191:6789/0 conn(0x7ff85c0a96b0 legacy=0x7ff85c1387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 1 monclient: found mon.a 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85c132980 con 0x7ff85c142a20 2024-03-25T18:14:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 5 monclient: authenticate success, global_id 4155 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 10 monclient: _renew_subs 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c13cb70 con 0x7ff85c142a20 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 10 client.4155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 10 monclient: _renew_subs 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c1390b0 con 0x7ff85c142a20 2024-03-25T18:14:09.466 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.463+0000 7ff86428d640 1 librados: init done 2024-03-25T18:14:09.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.465+0000 7ff86428d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.465+0000 7ff86428d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.465+0000 7ff86428d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.467 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.465+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c04f730 con 0x7ff85c142a20 2024-03-25T18:14:09.605 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.604+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff854001d30 con 0x7ff85c142a20 2024-03-25T18:14:09.605 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.604+0000 7ff852ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.605 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.604+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff854004d40 con 0x7ff85c142a20 2024-03-25T18:14:09.605 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.604+0000 7ff852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.605 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.604+0000 7ff852ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:09.605 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.604+0000 7ff852ffd640 10 monclient: dump: 2024-03-25T18:14:09.605 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.604+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff8540052f0 con 0x7ff85c142a20 2024-03-25T18:14:09.606 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.604+0000 7ff853fff640 4 set_mon_vals no callback set 2024-03-25T18:14:09.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.605+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1739627394 0 0) 0x7ff854006aa0 con 0x7ff85c142a20 2024-03-25T18:14:09.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.605+0000 7ff852ffd640 10 client.4155.objecter ms_dispatch 0x7ff85c000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:14:09.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.605+0000 7ff852ffd640 3 client.4155.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:14:09.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.605+0000 7ff852ffd640 3 client.4155.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:14:09.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.605+0000 7ff852ffd640 20 client.4155.objecter dump_active .. 0 homeless 2024-03-25T18:14:09.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.605+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff854005860 con 0x7ff85c142a20 2024-03-25T18:14:09.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.605+0000 7ff852ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.607 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.605+0000 7ff852ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:09.674 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:09.673+0000 7f49ca04c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:14:09.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.679+0000 7ff86428d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aad1d03e-a813-46d7-9e95-a932c7162531", "id": 6}] 2024-03-25T18:14:09.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.679+0000 7ff86428d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aad1d03e-a813-46d7-9e95-a932c7162531", "id": 6}] 2024-03-25T18:14:09.679 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.679+0000 7ff86428d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:09.680 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.679+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd new", "uuid": "aad1d03e-a813-46d7-9e95-a932c7162531", "id": 6} v 0) v1 -- 0x7ff85c0b77c0 con 0x7ff85c142a20 2024-03-25T18:14:09.708 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.707+0000 7ff852ffd640 1 -- 172.21.15.191:0/2411692634 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aad1d03e-a813-46d7-9e95-a932c7162531", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2853308910 0 1113811849) 0x7ff854006580 con 0x7ff85c142a20 2024-03-25T18:14:09.708 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.707+0000 7ff852ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aad1d03e-a813-46d7-9e95-a932c7162531", "id": 6}] 2024-03-25T18:14:09.708 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.707+0000 7ff852ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:09.708 INFO:teuthology.orchestra.run.smithi191.stdout:6 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 10 librados: watch_flush enter 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 10 librados: watch_flush exit 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 20 client.4155.objecter shutdown clearing up homeless session... 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 10 client.4155.objecter successfully canceled tick 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 10 monclient: shutdown 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 >> v1:172.21.15.162:6789/0 conn(0x7ff85c142a20 legacy=0x7ff85c132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 shutdown_connections 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 >> 172.21.15.191:0/2411692634 conn(0x7ff85c0668e0 msgr2=0x7ff85c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.708+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 shutdown_connections 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.709+0000 7ff86428d640 1 -- 172.21.15.191:0/2411692634 wait complete. 2024-03-25T18:14:09.710 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.709+0000 7ff86428d640 1 librados: shutdown 2024-03-25T18:14:09.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 96323cc0-8882-4860-b7f0-1e7fdc4e7d8c 7 2024-03-25T18:14:09.792 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:09.791+0000 7f214ce32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.822+0000 7feaf4295640 10 monclient: get_monmap_and_config 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.822+0000 7feaf4295640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.822+0000 7feaf4295640 10 monclient: monmap: 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:14:09.823408+0000 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:14:09.823408+0000 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:09.823 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:09.824 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:09.824 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.822+0000 7feaf4295640 1 Processor -- start 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 1 -- start start 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 10 monclient: init 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 10 monclient(hunting): picked mon.noname-a con 0x7feaec0b4700 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 10 monclient(hunting): picked mon.noname-b con 0x7feaec142a20 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 10 monclient(hunting): picked mon.noname-c con 0x7feaec0a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec0a9d60 con 0x7feaec0b4700 2024-03-25T18:14:09.825 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec13ff10 con 0x7feaec0a96b0 2024-03-25T18:14:09.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec0aa1f0 con 0x7feaec142a20 2024-03-25T18:14:09.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.823+0000 7feaf4295640 10 monclient(hunting): authenticate will time out at 7532.158626s 2024-03-25T18:14:09.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.824+0000 7feaf2832640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7feaec0a96b0 0x7feaec13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49788/0 (socket says 172.21.15.191:49788) 2024-03-25T18:14:09.826 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.824+0000 7feaf2832640 1 -- 172.21.15.191:0/2853465244 learned_addr learned my addr 172.21.15.191:0/2853465244 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:09.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.826+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348631522 0 0) 0x7feaec13ff10 con 0x7feaec0a96b0 2024-03-25T18:14:09.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.826+0000 7feaf102f640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.826+0000 7feaf102f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.826+0000 7feaf102f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.826+0000 7feaf102f640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:14:09.827 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.826+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead8003520 con 0x7feaec0a96b0 2024-03-25T18:14:09.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293503785 0 0) 0x7feaec0aa1f0 con 0x7feaec142a20 2024-03-25T18:14:09.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:14:09.828 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec13ff10 con 0x7feaec142a20 2024-03-25T18:14:09.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264228203 0 0) 0x7feaec0a9d60 con 0x7feaec0b4700 2024-03-25T18:14:09.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:14:09.829 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.827+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec0aa1f0 con 0x7feaec0b4700 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.829+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 154249391 0 0) 0x7fead8003520 con 0x7feaec0a96b0 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.829+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 >> v1:172.21.15.162:6790/0 conn(0x7feaec142a20 legacy=0x7feaec13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.829+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 >> v1:172.21.15.162:6789/0 conn(0x7feaec0b4700 legacy=0x7feaec142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 1 monclient: found mon.noname-c 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec0b4ae0 con 0x7feaec0a96b0 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7feae4001a10 con 0x7feaec0a96b0 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf4295640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf4295640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.831 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf102f640 10 monclient: dump: 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.832 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.830+0000 7feaf4295640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.831+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feae4001b70 con 0x7feaec0a96b0 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.831+0000 7feaf102f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.831+0000 7feaf102f640 1 -- 172.21.15.191:0/2853465244 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7feae4001f80 con 0x7feaec0a96b0 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.831+0000 7feaf102f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.831+0000 7feaf102f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.831+0000 7feaf102f640 10 monclient: dump: 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.831+0000 7feaf4295640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.831+0000 7feaf4295640 4 set_mon_vals no callback set 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.832+0000 7feaf4295640 10 monclient: shutdown 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.832+0000 7feaf4295640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.832+0000 7feaf4295640 1 -- 172.21.15.191:0/2853465244 >> v1:172.21.15.191:6789/0 conn(0x7feaec0a96b0 legacy=0x7feaec13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.832+0000 7feaf4295640 1 -- 172.21.15.191:0/2853465244 shutdown_connections 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.832+0000 7feaf4295640 1 -- 172.21.15.191:0/2853465244 >> 172.21.15.191:0/2853465244 conn(0x7feaec0668e0 msgr2=0x7feaec066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.832+0000 7feaf4295640 1 -- 172.21.15.191:0/2853465244 shutdown_connections 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.832+0000 7feaf4295640 1 -- 172.21.15.191:0/2853465244 wait complete. 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 10 monclient: build_initial_monmap 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 10 monclient: monmap: 2024-03-25T18:14:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 0 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 0.000000 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:created 0.000000 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 1 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 1 librados: starting msgr at 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 1 librados: starting objecter 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 1 Processor -- start 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 1 -- start start 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 1 librados: setting wanted keys 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 1 librados: calling monclient init 2024-03-25T18:14:09.836 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.833+0000 7feaf4295640 10 monclient: init 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 10 monclient(hunting): picked mon.noname-c con 0x7feaec0a96b0 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 10 monclient(hunting): picked mon.noname-a con 0x7feaec0b4700 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 10 monclient(hunting): picked mon.noname-b con 0x7feaec142a20 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec0af020 con 0x7feaec0b4700 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec0ae210 con 0x7feaec142a20 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaec0ae4f0 con 0x7feaec0a96b0 2024-03-25T18:14:09.838 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf4295640 10 monclient(hunting): authenticate will time out at 7532.169125s 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf2832640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7feaec142a20 0x7feaec1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:49790/0 (socket says 172.21.15.191:49790) 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.834+0000 7feaf2832640 1 -- 172.21.15.191:0/3765162812 learned_addr learned my addr 172.21.15.191:0/3765162812 (peer_addr_for_me v1:172.21.15.191:0/0) 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.835+0000 7feae2ffd640 10 client.?.objecter ms_handle_connect 0x7feaec142a20 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.835+0000 7feae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.835+0000 7feae2ffd640 10 client.?.objecter ms_handle_connect 0x7feaec0b4700 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.835+0000 7feae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.835+0000 7feae2ffd640 10 client.?.objecter ms_handle_connect 0x7feaec0a96b0 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.835+0000 7feae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501003110 0 0) 0x7feaec0ae210 con 0x7feaec142a20 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead0004eb0 con 0x7feaec142a20 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744377721 0 0) 0x7feaec0ae4f0 con 0x7feaec0a96b0 2024-03-25T18:14:09.839 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.837+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec0ae210 con 0x7feaec0a96b0 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.838+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747910359 0 0) 0x7feaec0af020 con 0x7feaec0b4700 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.838+0000 7feae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.838+0000 7feae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.838+0000 7feae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.838+0000 7feae2ffd640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:14:09.840 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.838+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec0ae4f0 con 0x7feaec0b4700 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.841+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3989912491 0 0) 0x7fead0004eb0 con 0x7feaec142a20 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaec0af020 con 0x7feaec142a20 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7feae4001ec0 con 0x7feaec142a20 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.843 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 555598229 0 0) 0x7feaec0ae210 con 0x7feaec0a96b0 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fead0004eb0 con 0x7feaec0a96b0 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7feadc001870 con 0x7feaec0a96b0 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.842+0000 7feae2ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.845 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.845 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.845 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.845 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.845 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.844+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3625987857 0 0) 0x7feaec0ae4f0 con 0x7feaec0b4700 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.844+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaec0ae210 con 0x7feaec0b4700 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.844+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7feae8001930 con 0x7feaec0b4700 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.844+0000 7feae2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.844+0000 7feae2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.844+0000 7feae2ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.846 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1541727883 0 0) 0x7fead0004eb0 con 0x7feaec0a96b0 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 >> v1:172.21.15.191:6789/0 conn(0x7feaec142a20 legacy=0x7feaec1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 >> v1:172.21.15.162:6789/0 conn(0x7feaec0b4700 legacy=0x7feaec0acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 1 monclient: found mon.c 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaec0ac0e0 con 0x7feaec0a96b0 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:09.847 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feae2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 5 monclient: authenticate success, global_id 4157 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 10 monclient: _renew_subs 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaec0ad340 con 0x7feaec0a96b0 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 10 monclient: _renew_subs 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feaec13b070 con 0x7feaec0a96b0 2024-03-25T18:14:09.848 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.845+0000 7feaf4295640 1 librados: init done 2024-03-25T18:14:09.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.846+0000 7feaf4295640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.846+0000 7feaf4295640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.846+0000 7feaf4295640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:09.849 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.846+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaec04f730 con 0x7feaec0a96b0 2024-03-25T18:14:09.944 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feadc0035f0 con 0x7feaec0a96b0 2024-03-25T18:14:09.944 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:09.944 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7feadc001d80 con 0x7feaec0a96b0 2024-03-25T18:14:09.944 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:09.944 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:09.944 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 10 monclient: dump: 2024-03-25T18:14:09.944 INFO:teuthology.orchestra.run.smithi191.stderr:epoch 1 2024-03-25T18:14:09.944 INFO:teuthology.orchestra.run.smithi191.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:election_strategy: 3 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7feadc003d10 con 0x7feaec0a96b0 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2479216440 0 0) 0x7feadc005530 con 0x7feaec0a96b0 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 10 client.4157.objecter ms_dispatch 0x7feaec000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 3 client.4157.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 3 client.4157.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feaf082e640 4 set_mon_vals no callback set 2024-03-25T18:14:09.945 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.943+0000 7feae2ffd640 20 client.4157.objecter dump_active .. 0 homeless 2024-03-25T18:14:09.946 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.944+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7feadc004fe0 con 0x7feaec0a96b0 2024-03-25T18:14:09.946 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.944+0000 7feae2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:09.946 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:09.944+0000 7feae2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:09.965 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:09.964+0000 7f49ca04c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:14:10.017 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:10.016+0000 7f49ca04c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:14:10.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.016+0000 7feaf4295640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "96323cc0-8882-4860-b7f0-1e7fdc4e7d8c", "id": 7}] 2024-03-25T18:14:10.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.016+0000 7feaf4295640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "96323cc0-8882-4860-b7f0-1e7fdc4e7d8c", "id": 7}] 2024-03-25T18:14:10.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.016+0000 7feaf4295640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:10.018 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.016+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd new", "uuid": "96323cc0-8882-4860-b7f0-1e7fdc4e7d8c", "id": 7} v 0) v1 -- 0x7feaec13b320 con 0x7feaec0a96b0 2024-03-25T18:14:10.095 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:10.094+0000 7f214ce32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:14:10.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.100+0000 7feae2ffd640 1 -- 172.21.15.191:0/3765162812 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "96323cc0-8882-4860-b7f0-1e7fdc4e7d8c", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 276519095 0 1371665406) 0x7feadc005300 con 0x7feaec0a96b0 2024-03-25T18:14:10.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.100+0000 7feae2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "96323cc0-8882-4860-b7f0-1e7fdc4e7d8c", "id": 7}] 2024-03-25T18:14:10.101 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.100+0000 7feae2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:10.101 INFO:teuthology.orchestra.run.smithi191.stdout:7 2024-03-25T18:14:10.102 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 10 librados: watch_flush enter 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 10 librados: watch_flush exit 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 20 client.4157.objecter shutdown clearing up homeless session... 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 10 client.4157.objecter successfully canceled tick 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 10 monclient: shutdown 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 >> v1:172.21.15.162:6790/0 conn(0x7feaec0a96b0 legacy=0x7feaec0ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 shutdown_connections 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 >> 172.21.15.191:0/3765162812 conn(0x7feaec0668e0 msgr2=0x7feaec0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 shutdown_connections 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 1 -- 172.21.15.191:0/3765162812 wait complete. 2024-03-25T18:14:10.103 INFO:teuthology.orchestra.run.smithi191.stderr:2024-03-25T18:14:10.101+0000 7feaf4295640 1 librados: shutdown 2024-03-25T18:14:10.112 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:10.113 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:14:10.113 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:14:10.115 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:14:10.115 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:10.115 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:14:10.115 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:14:10.117 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:14:10.117 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:10.117 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:14:10.118 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:14:10.119 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:14:10.120 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:10.120 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:14:10.120 DEBUG:teuthology.orchestra.run.smithi162:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:14:10.122 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:14:10.122 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:10.122 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:14:10.122 DEBUG:teuthology.orchestra.run.smithi191:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:14:10.125 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:14:10.125 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:10.125 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:14:10.125 DEBUG:teuthology.orchestra.run.smithi191:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:14:10.127 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:14:10.127 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:10.127 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:14:10.127 DEBUG:teuthology.orchestra.run.smithi191:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:14:10.129 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:14:10.129 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:10.129 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:14:10.130 DEBUG:teuthology.orchestra.run.smithi191:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:14:10.131 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:14:10.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:10.141 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:10.140+0000 7f49ca04c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:14:10.161 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:10.160+0000 7f214ce32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.297+0000 7fea91796640 10 monclient: get_monmap_and_config 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.297+0000 7fea91796640 10 monclient: build_initial_monmap 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.298+0000 7fea91796640 10 monclient: monmap: 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:14:10.299036+0000 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:14:10.299036+0000 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:10.299 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.298+0000 7fea91796640 1 Processor -- start 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.299+0000 7fea91796640 1 -- start start 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.299+0000 7fea91796640 10 monclient: init 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.299+0000 7fea91796640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.299+0000 7fea91796640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.299+0000 7fea91796640 10 monclient(hunting): picked mon.noname-b con 0x7fea8c0b4700 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.299+0000 7fea91796640 10 monclient(hunting): picked mon.noname-a con 0x7fea8c142a20 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.299+0000 7fea91796640 10 monclient(hunting): picked mon.noname-c con 0x7fea8c0a96b0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.299+0000 7fea91796640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c0a9d60 con 0x7fea8c142a20 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.300+0000 7fea91796640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c13ff10 con 0x7fea8c0a96b0 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.300+0000 7fea91796640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c0aa1f0 con 0x7fea8c0b4700 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.300+0000 7fea91796640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:10.301 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.300+0000 7fea91796640 10 monclient(hunting): authenticate will time out at 7549.573670s 2024-03-25T18:14:10.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.301+0000 7fea8affd640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fea8c0b4700 0x7fea8c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:52764/0 (socket says 172.21.15.162:52764) 2024-03-25T18:14:10.302 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.301+0000 7fea8affd640 1 -- 172.21.15.162:0/3434921677 learned_addr learned my addr 172.21.15.162:0/3434921677 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:10.352 INFO:tasks.ceph.mgr.y.smithi162.stderr:2024-03-25T18:14:10.352+0000 7f214ce32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:14:10.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.414+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150146528 0 0) 0x7fea8c13ff10 con 0x7fea8c0a96b0 2024-03-25T18:14:10.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.414+0000 7fea89ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:10.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.414+0000 7fea89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:10.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.414+0000 7fea89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:10.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.414+0000 7fea89ffb640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:14:10.415 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.414+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea78003520 con 0x7fea8c0a96b0 2024-03-25T18:14:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869301462 0 0) 0x7fea78003520 con 0x7fea8c0a96b0 2024-03-25T18:14:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 >> v1:172.21.15.162:6790/0 conn(0x7fea8c0b4700 legacy=0x7fea8c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 >> v1:172.21.15.162:6789/0 conn(0x7fea8c142a20 legacy=0x7fea8c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c0b4ae0 con 0x7fea8c0a96b0 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fea84001a10 con 0x7fea8c0a96b0 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea89ffb640 10 monclient: dump: 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:10.421 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:10.422 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:10.422 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:10.422 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:10.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea91796640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:14:10.422 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.419+0000 7fea91796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.445+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea84001be0 con 0x7fea8c0a96b0 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.445+0000 7fea89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.445+0000 7fea89ffb640 1 -- 172.21.15.162:0/3434921677 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fea84002030 con 0x7fea8c0a96b0 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.445+0000 7fea89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.445+0000 7fea89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.445+0000 7fea89ffb640 10 monclient: dump: 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:10.446 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.447 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:10.447 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:10.447 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:10.447 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:10.447 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:10.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.445+0000 7fea91796640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:10.447 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.445+0000 7fea91796640 4 set_mon_vals no callback set 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.446+0000 7fea91796640 10 monclient: shutdown 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.446+0000 7fea91796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.446+0000 7fea91796640 1 -- 172.21.15.162:0/3434921677 >> v1:172.21.15.191:6789/0 conn(0x7fea8c0a96b0 legacy=0x7fea8c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.446+0000 7fea91796640 1 -- 172.21.15.162:0/3434921677 shutdown_connections 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.446+0000 7fea91796640 1 -- 172.21.15.162:0/3434921677 >> 172.21.15.162:0/3434921677 conn(0x7fea8c0668e0 msgr2=0x7fea8c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.446+0000 7fea91796640 1 -- 172.21.15.162:0/3434921677 shutdown_connections 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.446+0000 7fea91796640 1 -- 172.21.15.162:0/3434921677 wait complete. 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.447+0000 7fea91796640 10 monclient: build_initial_monmap 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.447+0000 7fea91796640 10 monclient: monmap: 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 0 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:created 0.000000 2024-03-25T18:14:10.448 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:10.449 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:10.449 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:10.449 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:10.449 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:10.449 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:10.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.447+0000 7fea91796640 1 librados: starting msgr at 2024-03-25T18:14:10.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.447+0000 7fea91796640 1 librados: starting objecter 2024-03-25T18:14:10.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.448+0000 7fea91796640 1 Processor -- start 2024-03-25T18:14:10.450 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.448+0000 7fea91796640 1 -- start start 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.448+0000 7fea91796640 1 librados: setting wanted keys 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.448+0000 7fea91796640 1 librados: calling monclient init 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.448+0000 7fea91796640 10 monclient: init 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 10 monclient(hunting): picked mon.noname-a con 0x7fea8c0a96b0 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 10 monclient(hunting): picked mon.noname-c con 0x7fea8c0b4700 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 10 monclient(hunting): picked mon.noname-b con 0x7fea8c142a20 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c0afeb0 con 0x7fea8c0a96b0 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c1335d0 con 0x7fea8c142a20 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c1338b0 con 0x7fea8c0b4700 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:10.451 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.449+0000 7fea91796640 10 monclient(hunting): authenticate will time out at 7549.723507s 2024-03-25T18:14:10.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.450+0000 7fea8b7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fea8c142a20 0x7fea8c1e6850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:34768/0 (socket says 172.21.15.162:34768) 2024-03-25T18:14:10.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.450+0000 7fea8b7fe640 1 -- 172.21.15.162:0/2976116020 learned_addr learned my addr 172.21.15.162:0/2976116020 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:10.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.450+0000 7fea837fe640 10 client.?.objecter ms_handle_connect 0x7fea8c142a20 2024-03-25T18:14:10.452 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.450+0000 7fea837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:10.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.453+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693965447 0 0) 0x7fea8c1335d0 con 0x7fea8c142a20 2024-03-25T18:14:10.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.453+0000 7fea837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:10.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.453+0000 7fea837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:10.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.453+0000 7fea837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:10.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.453+0000 7fea837fe640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:14:10.454 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.453+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c004eb0 con 0x7fea8c142a20 2024-03-25T18:14:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.463+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070615008 0 0) 0x7fea6c004eb0 con 0x7fea8c142a20 2024-03-25T18:14:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.463+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea8c1335d0 con 0x7fea8c142a20 2024-03-25T18:14:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.463+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fea84001f30 con 0x7fea8c142a20 2024-03-25T18:14:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.463+0000 7fea837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.463+0000 7fea837fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.463+0000 7fea837fe640 10 monclient(hunting): dump: 2024-03-25T18:14:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-03-25T18:14:10.464 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:10.465 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.463+0000 7fea837fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:10.470 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4137481645 0 0) 0x7fea8c1335d0 con 0x7fea8c142a20 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 >> v1:172.21.15.162:6790/0 conn(0x7fea8c0b4700 legacy=0x7fea8c0ae440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 >> v1:172.21.15.162:6789/0 conn(0x7fea8c0a96b0 legacy=0x7fea8c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 1 monclient: found mon.b 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c130850 con 0x7fea8c142a20 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.469+0000 7fea837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 5 monclient: authenticate success, global_id 4159 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 10 monclient: _renew_subs 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8c0aed50 con 0x7fea8c142a20 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:10.471 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 10 monclient: _renew_subs 2024-03-25T18:14:10.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:10.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8c0ada30 con 0x7fea8c142a20 2024-03-25T18:14:10.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.470+0000 7fea91796640 1 librados: init done 2024-03-25T18:14:10.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.471+0000 7fea91796640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:10.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.471+0000 7fea91796640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:10.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.471+0000 7fea91796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:10.472 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.471+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea8c0ae160 con 0x7fea8c142a20 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.493+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea84002090 con 0x7fea8c142a20 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.493+0000 7fea837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.493+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fea84003c60 con 0x7fea8c142a20 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.493+0000 7fea837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.493+0000 7fea837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.493+0000 7fea837fe640 10 monclient: dump: 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:epoch 1 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:10.494 INFO:teuthology.orchestra.run.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.493+0000 7fea897fa640 4 set_mon_vals no callback set 2024-03-25T18:14:10.495 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.493+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1243986843 0 0) 0x7fea84003e80 con 0x7fea8c142a20 2024-03-25T18:14:10.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.496+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fea84015a40 con 0x7fea8c142a20 2024-03-25T18:14:10.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.496+0000 7fea837fe640 10 client.4159.objecter ms_dispatch 0x7fea8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:10.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.496+0000 7fea837fe640 3 client.4159.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:10.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.496+0000 7fea837fe640 3 client.4159.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:10.497 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.496+0000 7fea837fe640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-25T18:14:10.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.674+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fea84014ee0 con 0x7fea8c142a20 2024-03-25T18:14:10.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.674+0000 7fea837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:10.675 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.674+0000 7fea837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:10.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.782+0000 7fea91796640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.782+0000 7fea91796640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.782+0000 7fea91796640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:10.783 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.782+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea8c04f730 con 0x7fea8c142a20 2024-03-25T18:14:10.921 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.920+0000 7fea837fe640 1 -- 172.21.15.162:0/2976116020 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fea840134e0 con 0x7fea8c142a20 2024-03-25T18:14:10.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.920+0000 7fea837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.922 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.920+0000 7fea837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:10.922 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-25T18:14:10.922 INFO:teuthology.orchestra.run.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:10.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 10 librados: watch_flush enter 2024-03-25T18:14:10.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 10 librados: watch_flush exit 2024-03-25T18:14:10.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-25T18:14:10.923 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 10 client.4159.objecter successfully canceled tick 2024-03-25T18:14:10.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 10 monclient: shutdown 2024-03-25T18:14:10.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:10.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 >> v1:172.21.15.191:6789/0 conn(0x7fea8c142a20 legacy=0x7fea8c1e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 shutdown_connections 2024-03-25T18:14:10.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 >> 172.21.15.162:0/2976116020 conn(0x7fea8c0668e0 msgr2=0x7fea8c0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:10.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 shutdown_connections 2024-03-25T18:14:10.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 1 -- 172.21.15.162:0/2976116020 wait complete. 2024-03-25T18:14:10.924 INFO:teuthology.orchestra.run.smithi162.stderr:2024-03-25T18:14:10.922+0000 7fea91796640 1 librados: shutdown 2024-03-25T18:14:10.938 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:14:10.938 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:14:11.703 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:11.702+0000 7f496ad11640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:14:11.744 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:11.776 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:14:11.775+0000 7f496ad11640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.886+0000 7f38d7d5d640 10 monclient: get_monmap_and_config 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.886+0000 7f38d7d5d640 10 monclient: build_initial_monmap 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.886+0000 7f38d7d5d640 10 monclient: monmap: 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:14:11.887452+0000 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:14:11.887452+0000 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:11.887 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:11.888 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:11.888 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:11.888 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:11.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.887+0000 7f38d7d5d640 1 Processor -- start 2024-03-25T18:14:11.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.887+0000 7f38d7d5d640 1 -- start start 2024-03-25T18:14:11.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.887+0000 7f38d7d5d640 10 monclient: init 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f38d00a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f38d013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f38d01430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d0142520 con 0x7f38d01430c0 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d00b28a0 con 0x7f38d00a98f0 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d00b2a30 con 0x7f38d013f710 2024-03-25T18:14:11.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:11.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.888+0000 7f38d7d5d640 10 monclient(hunting): authenticate will time out at 7551.162339s 2024-03-25T18:14:11.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.889+0000 7f38d52f8640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f38d013f710 0x7f38d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:52776/0 (socket says 172.21.15.162:52776) 2024-03-25T18:14:11.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:11.889+0000 7f38d52f8640 1 -- 172.21.15.162:0/4225035015 learned_addr learned my addr 172.21.15.162:0/4225035015 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:12.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.022+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599132523 0 0) 0x7f38d00b28a0 con 0x7f38d00a98f0 2024-03-25T18:14:12.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.022+0000 7f38d4af7640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.022+0000 7f38d4af7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.022+0000 7f38d4af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.024 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.022+0000 7f38d4af7640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:14:12.024 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.022+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38bc003520 con 0x7f38d00a98f0 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2404770519 0 0) 0x7f38bc003520 con 0x7f38d00a98f0 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 >> v1:172.21.15.162:6790/0 conn(0x7f38d013f710 legacy=0x7f38d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 >> v1:172.21.15.162:6789/0 conn(0x7f38d01430c0 legacy=0x7f38d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 10 monclient: _finish_hunting 0 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 1 monclient: found mon.noname-c 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d00b44c0 con 0x7f38d00a98f0 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 10 monclient: _finish_auth 0 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 10 monclient: _check_auth_tickets 2024-03-25T18:14:12.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d7d5d640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d7d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f38cc001690 con 0x7f38d00a98f0 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d4af7640 10 monclient: dump: 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.028+0000 7f38d7d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.030+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38cc001920 con 0x7f38d00a98f0 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.030+0000 7f38d4af7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.030+0000 7f38d7d5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.030+0000 7f38d7d5d640 4 set_mon_vals no callback set 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d4af7640 1 -- 172.21.15.162:0/4225035015 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f38cc001c30 con 0x7f38d00a98f0 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d4af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d4af7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d4af7640 10 monclient: dump: 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d7d5d640 10 monclient: shutdown 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d7d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d7d5d640 1 -- 172.21.15.162:0/4225035015 >> v1:172.21.15.191:6789/0 conn(0x7f38d00a98f0 legacy=0x7f38d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d7d5d640 1 -- 172.21.15.162:0/4225035015 shutdown_connections 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d7d5d640 1 -- 172.21.15.162:0/4225035015 >> 172.21.15.162:0/4225035015 conn(0x7f38d00b7ea0 msgr2=0x7f38d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d7d5d640 1 -- 172.21.15.162:0/4225035015 shutdown_connections 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.031+0000 7f38d7d5d640 1 -- 172.21.15.162:0/4225035015 wait complete. 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.032+0000 7f38d7d5d640 10 monclient: build_initial_monmap 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.032+0000 7f38d7d5d640 10 monclient: monmap: 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.033+0000 7f38d7d5d640 1 librados: starting msgr at 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.033+0000 7f38d7d5d640 1 librados: starting objecter 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.033+0000 7f38d7d5d640 1 Processor -- start 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.033+0000 7f38d7d5d640 1 -- start start 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.033+0000 7f38d7d5d640 1 librados: setting wanted keys 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.033+0000 7f38d7d5d640 1 librados: calling monclient init 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.033+0000 7f38d7d5d640 10 monclient: init 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f38d00a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f38d013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f38d01430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d0070820 con 0x7f38d00a98f0 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d01dbbd0 con 0x7f38d01430c0 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d006f600 con 0x7f38d013f710 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.034+0000 7f38d7d5d640 10 monclient(hunting): authenticate will time out at 7551.308184s 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.035+0000 7f38d52f8640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f38d013f710 0x7f38d01dc0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:52788/0 (socket says 172.21.15.162:52788) 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.035+0000 7f38d52f8640 1 -- 172.21.15.162:0/1084768708 learned_addr learned my addr 172.21.15.162:0/1084768708 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.035+0000 7f38c67fc640 10 client.?.objecter ms_handle_connect 0x7f38d013f710 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.035+0000 7f38c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.035+0000 7f38c67fc640 10 client.?.objecter ms_handle_connect 0x7f38d01430c0 2024-03-25T18:14:12.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.035+0000 7f38c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.039 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.038+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817302149 0 0) 0x7f38d01dbbd0 con 0x7f38d01430c0 2024-03-25T18:14:12.039 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.038+0000 7f38c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.038+0000 7f38c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.038+0000 7f38c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.038+0000 7f38c67fc640 10 monclient(hunting): my global_id is 4165 2024-03-25T18:14:12.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.038+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b4004eb0 con 0x7f38d01430c0 2024-03-25T18:14:12.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.044+0000 7f38c67fc640 10 client.?.objecter ms_handle_connect 0x7f38d00a98f0 2024-03-25T18:14:12.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.044+0000 7f38c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.044+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817432225 0 0) 0x7f38d006f600 con 0x7f38d013f710 2024-03-25T18:14:12.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.044+0000 7f38c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.044+0000 7f38c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.044+0000 7f38c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.044+0000 7f38c67fc640 10 monclient(hunting): my global_id is 4175 2024-03-25T18:14:12.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.044+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d01dbbd0 con 0x7f38d013f710 2024-03-25T18:14:12.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.046+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 824856577 0 0) 0x7f38b4004eb0 con 0x7f38d01430c0 2024-03-25T18:14:12.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.046+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d006f600 con 0x7f38d01430c0 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.046+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f38c8001be0 con 0x7f38d01430c0 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.046+0000 7f38c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.046+0000 7f38c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.046+0000 7f38c67fc640 10 monclient(hunting): dump: 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:12.047 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:12.048 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:12.048 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:12.048 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:12.048 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.046+0000 7f38c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:12.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.050+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392325227 0 0) 0x7f38d0070820 con 0x7f38d00a98f0 2024-03-25T18:14:12.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.051+0000 7f38c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.051+0000 7f38c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.051+0000 7f38c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.051+0000 7f38c67fc640 10 monclient(hunting): my global_id is 4179 2024-03-25T18:14:12.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.051+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b4004eb0 con 0x7f38d00a98f0 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3087636236 0 0) 0x7f38d01dbbd0 con 0x7f38d013f710 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d0070820 con 0x7f38d013f710 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f38c00018f0 con 0x7f38d013f710 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 10 monclient(hunting): dump: 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.054 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1555585697 0 0) 0x7f38d006f600 con 0x7f38d01430c0 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 >> v1:172.21.15.162:6790/0 conn(0x7f38d013f710 legacy=0x7f38d01dc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 >> v1:172.21.15.162:6789/0 conn(0x7f38d00a98f0 legacy=0x7f38d006eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 1 monclient: found mon.b 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d01dc800 con 0x7f38d01430c0 2024-03-25T18:14:12.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38d7d5d640 5 monclient: authenticate success, global_id 4165 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38d7d5d640 10 monclient: _renew_subs 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38d7d5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d006e680 con 0x7f38d01430c0 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.052+0000 7f38d7d5d640 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.053+0000 7f38d7d5d640 10 monclient: _renew_subs 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.053+0000 7f38d7d5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.053+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d0070db0 con 0x7f38d01430c0 2024-03-25T18:14:12.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.053+0000 7f38d7d5d640 1 librados: init done 2024-03-25T18:14:12.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.054+0000 7f38d7d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.054+0000 7f38d7d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.054+0000 7f38d7d5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:12.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.054+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d01dc9f0 con 0x7f38d01430c0 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.083+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38c80020b0 con 0x7f38d01430c0 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.083+0000 7f38c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.083+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f38c8003970 con 0x7f38d01430c0 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.083+0000 7f38c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.083+0000 7f38c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.083+0000 7f38c67fc640 10 monclient: dump: 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.083+0000 7f38c7fff640 4 set_mon_vals no callback set 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.084+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3557445347 0 0) 0x7f38c80198d0 con 0x7f38d01430c0 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.085+0000 7f38d5af9640 10 monclient: get_auth_request con 0x7f38b406de20 auth_method 0 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.086+0000 7f38c67fc640 10 client.4165.objecter ms_handle_connect 0x7f38b406de20 2024-03-25T18:14:12.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.087+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f38c8053670 con 0x7f38d01430c0 2024-03-25T18:14:12.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.088+0000 7f38c67fc640 10 client.4165.objecter ms_dispatch 0x7f38d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:12.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.088+0000 7f38c67fc640 3 client.4165.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:12.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.088+0000 7f38c67fc640 3 client.4165.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:12.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.088+0000 7f38c67fc640 20 client.4165.objecter dump_active .. 0 homeless 2024-03-25T18:14:12.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.504+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38c804f180 con 0x7f38d01430c0 2024-03-25T18:14:12.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.504+0000 7f38c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.504+0000 7f38c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:12.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.841+0000 7f38d7d5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:12.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.841+0000 7f38d7d5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:12.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.841+0000 7f38d7d5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:12.843 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.841+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38d00627b0 con 0x7f38d01430c0 2024-03-25T18:14:12.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.866+0000 7f38c67fc640 1 -- 172.21.15.162:0/1084768708 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f38c80510b0 con 0x7f38d01430c0 2024-03-25T18:14:12.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.866+0000 7f38c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:12.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.866+0000 7f38c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:12.867 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:14:12.867 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:12.870 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 10 librados: watch_flush enter 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 10 librados: watch_flush exit 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 20 client.4165.objecter shutdown clearing up homeless session... 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 10 client.4165.objecter successfully canceled tick 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 >> v1:172.21.15.191:6800/1552703795 conn(0x7f38b406de20 legacy=0x7f38b40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 10 monclient: shutdown 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 >> v1:172.21.15.191:6789/0 conn(0x7f38d01430c0 legacy=0x7f38d01e6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 shutdown_connections 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.869+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 >> 172.21.15.162:0/1084768708 conn(0x7f38d00b7ea0 msgr2=0x7f38d013ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.870+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 shutdown_connections 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.870+0000 7f38d7d5d640 1 -- 172.21.15.162:0/1084768708 wait complete. 2024-03-25T18:14:12.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:12.870+0000 7f38d7d5d640 1 librados: shutdown 2024-03-25T18:14:12.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:17.029 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:06.837 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.067 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:06.871 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.090 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:06.898 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.100 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:06.907 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.135 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:06.939 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.174 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:06.981 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.423 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:07.212 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.424 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:07.219 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.430 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:07.240 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.457 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:07.270 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.488 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:07.299 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.492 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:07.287 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.497 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:07.287 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.542 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:07.355 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.638 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-25T18:14:17.629+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:17.717 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-25T18:14:17.710+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:17.731 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-25T18:14:17.723+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:17.792 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:07.587 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:17.851 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:07.646 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.033 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-03-25T18:14:18.026+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:18.040 INFO:tasks.ceph.osd.3.smithi162.stderr:2024-03-25T18:14:18.031+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:18.070 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-25T18:14:18.063+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:18.090 INFO:tasks.ceph.osd.0.smithi162.stderr:2024-03-25T18:14:18.084+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:18.094 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:07.903 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.194 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:07.998 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.212 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:08.019 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.449 INFO:tasks.ceph.osd.1.smithi162.stderr:2024-03-25T18:14:18.440+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:18.461 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:08.269 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.494 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:08.289 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.504 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:08.293 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.555 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:08.365 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.581 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:08.388 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.586 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:08.390 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.625 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:08.437 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.776 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:08.584 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.863 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:08.658 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.884 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:08.673 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.899 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:08.706 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.919 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:08.730 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.949 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:08.753 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.961 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:08.756 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:18.991 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:08.804 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.049 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:08.857 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.097 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:08.908 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.174 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:08.981 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.199 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:08.988 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.208 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:09.003 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.245 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:09.049 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.284 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:09.095 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.313 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:09.121 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.322 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:09.117 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.373 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:09.185 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.452 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:09.259 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.464 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:09.253 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.481 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:09.276 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.538 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:09.341 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.584 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:09.394 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.590 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:09.398 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.633 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:09.428 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.662 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:09.474 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.690 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:19.716 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:09.523 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.738 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:09.527 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.763 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:09.558 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.779+0000 7fb83482a640 10 monclient: get_monmap_and_config 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.779+0000 7fb83482a640 10 monclient: build_initial_monmap 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.779+0000 7fb83482a640 10 monclient: monmap: 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:14:19.780268+0000 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:14:19.780268+0000 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:19.780 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.779+0000 7fb83482a640 1 Processor -- start 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.779+0000 7fb83482a640 1 -- start start 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.779+0000 7fb83482a640 10 monclient: init 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 10 monclient(hunting): picked mon.noname-b con 0x7fb8300a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 10 monclient(hunting): picked mon.noname-a con 0x7fb83013f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 10 monclient(hunting): picked mon.noname-c con 0x7fb8301430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb830142520 con 0x7fb83013f710 2024-03-25T18:14:19.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8300b28a0 con 0x7fb8301430c0 2024-03-25T18:14:19.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8300b2a30 con 0x7fb8300a98f0 2024-03-25T18:14:19.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:19.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.780+0000 7fb83482a640 10 monclient(hunting): authenticate will time out at 7559.054216s 2024-03-25T18:14:19.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.781+0000 7fb82e59c640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fb8300a98f0 0x7fb8300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43328/0 (socket says 172.21.15.162:43328) 2024-03-25T18:14:19.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.781+0000 7fb82e59c640 1 -- 172.21.15.162:0/1066452113 learned_addr learned my addr 172.21.15.162:0/1066452113 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.782+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603446623 0 0) 0x7fb8300b2a30 con 0x7fb8300a98f0 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb818003520 con 0x7fb8300a98f0 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983129528 0 0) 0x7fb830142520 con 0x7fb83013f710 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:14:19.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8300b2a30 con 0x7fb83013f710 2024-03-25T18:14:19.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344600778 0 0) 0x7fb8300b28a0 con 0x7fb8301430c0 2024-03-25T18:14:19.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 10 monclient(hunting): my global_id is 4192 2024-03-25T18:14:19.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.783+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb830142520 con 0x7fb8301430c0 2024-03-25T18:14:19.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129864935 0 0) 0x7fb818003520 con 0x7fb8300a98f0 2024-03-25T18:14:19.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 >> v1:172.21.15.191:6789/0 conn(0x7fb8301430c0 legacy=0x7fb83014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 >> v1:172.21.15.162:6789/0 conn(0x7fb83013f710 legacy=0x7fb8301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 1 monclient: found mon.noname-b 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8300b44c0 con 0x7fb8300a98f0 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 10 monclient: _finish_auth 0 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb828001690 con 0x7fb8300a98f0 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb82d59a640 10 monclient: dump: 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:19.788 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb83482a640 5 monclient: authenticate success, global_id 4202 2024-03-25T18:14:19.789 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.786+0000 7fb83482a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8280017f0 con 0x7fb8300a98f0 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb82d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb82d59a640 1 -- 172.21.15.162:0/1066452113 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb828001c40 con 0x7fb8300a98f0 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb83482a640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb83482a640 4 set_mon_vals no callback set 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb82d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb82d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb82d59a640 10 monclient: dump: 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:19.791 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb83482a640 10 monclient: shutdown 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb83482a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.788+0000 7fb83482a640 1 -- 172.21.15.162:0/1066452113 >> v1:172.21.15.162:6790/0 conn(0x7fb8300a98f0 legacy=0x7fb8300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.789+0000 7fb83482a640 1 -- 172.21.15.162:0/1066452113 shutdown_connections 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.789+0000 7fb83482a640 1 -- 172.21.15.162:0/1066452113 >> 172.21.15.162:0/1066452113 conn(0x7fb8300b7ea0 msgr2=0x7fb8300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.789+0000 7fb83482a640 1 -- 172.21.15.162:0/1066452113 shutdown_connections 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.789+0000 7fb83482a640 1 -- 172.21.15.162:0/1066452113 wait complete. 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 10 monclient: build_initial_monmap 2024-03-25T18:14:19.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 10 monclient: monmap: 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 1 librados: starting msgr at 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 1 librados: starting objecter 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 1 Processor -- start 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 1 -- start start 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 1 librados: setting wanted keys 2024-03-25T18:14:19.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 1 librados: calling monclient init 2024-03-25T18:14:19.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 10 monclient: init 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.790+0000 7fb83482a640 10 monclient(hunting): picked mon.noname-b con 0x7fb8300a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb83482a640 10 monclient(hunting): picked mon.noname-a con 0x7fb83013f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb83482a640 10 monclient(hunting): picked mon.noname-c con 0x7fb8301430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb83482a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8301384a0 con 0x7fb83013f710 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb83482a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb830139940 con 0x7fb8300a98f0 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb83482a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8301389c0 con 0x7fb8301430c0 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb83482a640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb83482a640 10 monclient(hunting): authenticate will time out at 7559.064702s 2024-03-25T18:14:19.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb82ed9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fb8301430c0 0x7fb8301e1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43330/0 (socket says 172.21.15.162:43330) 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb82ed9d640 1 -- 172.21.15.162:0/4113090576 learned_addr learned my addr 172.21.15.162:0/4113090576 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb81effd640 10 client.?.objecter ms_handle_connect 0x7fb8301430c0 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb81effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb81effd640 10 client.?.objecter ms_handle_connect 0x7fb8300a98f0 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb81effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb81effd640 10 client.?.objecter ms_handle_connect 0x7fb83013f710 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.791+0000 7fb81effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077311927 0 0) 0x7fb830139940 con 0x7fb8300a98f0 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 10 monclient(hunting): my global_id is 4195 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb810004eb0 con 0x7fb8300a98f0 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515097755 0 0) 0x7fb8301384a0 con 0x7fb83013f710 2024-03-25T18:14:19.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.793+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb830139940 con 0x7fb83013f710 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.794+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484257703 0 0) 0x7fb8301389c0 con 0x7fb8301430c0 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.794+0000 7fb81effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.794+0000 7fb81effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.794+0000 7fb81effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.794+0000 7fb81effd640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:14:19.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.794+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8301384a0 con 0x7fb8301430c0 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3419588834 0 0) 0x7fb810004eb0 con 0x7fb8300a98f0 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8301389c0 con 0x7fb8300a98f0 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb828001b00 con 0x7fb8300a98f0 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 10 monclient(hunting): dump: 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:19.800 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4127604496 0 0) 0x7fb830139940 con 0x7fb83013f710 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb810004eb0 con 0x7fb83013f710 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb824001c70 con 0x7fb83013f710 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.798+0000 7fb81effd640 10 monclient(hunting): dump: 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.801 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.799+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2555087627 0 0) 0x7fb8301384a0 con 0x7fb8301430c0 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.799+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb830139940 con 0x7fb8301430c0 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.799+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb820001850 con 0x7fb8301430c0 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.799+0000 7fb81effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.799+0000 7fb81effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.799+0000 7fb81effd640 10 monclient(hunting): dump: 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:19.802 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.803 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.803 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.803 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:19.803 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:19.803 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:19.803 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:19.803 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:19.804 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 571373335 0 0) 0x7fb8301389c0 con 0x7fb8300a98f0 2024-03-25T18:14:19.804 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 >> v1:172.21.15.162:6790/0 conn(0x7fb8301430c0 legacy=0x7fb8301e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.804 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 >> v1:172.21.15.162:6789/0 conn(0x7fb83013f710 legacy=0x7fb8301da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.804 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:19.804 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 1 monclient: found mon.b 2024-03-25T18:14:19.804 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:19.804 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:19.804 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8300b1660 con 0x7fb8300a98f0 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 10 monclient: _finish_auth 0 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb81effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 5 monclient: authenticate success, global_id 4195 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 10 monclient: _renew_subs 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb830137ff0 con 0x7fb8300a98f0 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 10 monclient: _renew_subs 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8301db170 con 0x7fb8300a98f0 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.801+0000 7fb83482a640 1 librados: init done 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.802+0000 7fb83482a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.802+0000 7fb83482a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.802+0000 7fb83482a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:19.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.802+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8300b3ef0 con 0x7fb8300a98f0 2024-03-25T18:14:19.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.817+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb828001c80 con 0x7fb8300a98f0 2024-03-25T18:14:19.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.817+0000 7fb81effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:19.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.817+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb828004d90 con 0x7fb8300a98f0 2024-03-25T18:14:19.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.817+0000 7fb81effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.817+0000 7fb81effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:19.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.817+0000 7fb81effd640 10 monclient: dump: 2024-03-25T18:14:19.818 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:19.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.817+0000 7fb82cd99640 4 set_mon_vals no callback set 2024-03-25T18:14:19.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.818+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fb82801ad60 con 0x7fb8300a98f0 2024-03-25T18:14:19.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.818+0000 7fb82dd9b640 10 monclient: get_auth_request con 0x7fb81006e0f0 auth_method 0 2024-03-25T18:14:19.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.819+0000 7fb81effd640 10 client.4195.objecter ms_handle_connect 0x7fb81006e0f0 2024-03-25T18:14:19.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.820+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fb828054870 con 0x7fb8300a98f0 2024-03-25T18:14:19.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.820+0000 7fb81effd640 10 client.4195.objecter ms_dispatch 0x7fb830000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:19.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.820+0000 7fb81effd640 3 client.4195.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:19.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.820+0000 7fb81effd640 3 client.4195.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:19.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:19.820+0000 7fb81effd640 20 client.4195.objecter dump_active .. 0 homeless 2024-03-25T18:14:19.838 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:09.642 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.864 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:09.672 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.882 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:09.693 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.903 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:09.698 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.952 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:09.764 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:19.981 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:09.788 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.011 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:09.800 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.034 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:09.829 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.050+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb828050380 con 0x7fb8300a98f0 2024-03-25T18:14:20.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.050+0000 7fb81effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:20.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.050+0000 7fb81effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:20.130 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:09.934 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.133 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:09.941 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.171 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:09.966 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.174 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:09.984 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.223+0000 7fb83482a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:20.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.223+0000 7fb83482a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:20.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.223+0000 7fb83482a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:20.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.223+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb830138630 con 0x7fb8300a98f0 2024-03-25T18:14:20.238 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.237+0000 7fb81effd640 1 -- 172.21.15.162:0/4113090576 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fb8280522b0 con 0x7fb8300a98f0 2024-03-25T18:14:20.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.238+0000 7fb81effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:20.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.238+0000 7fb81effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:20.239 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:14:20.239 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:20.239 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:10.052 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 10 librados: watch_flush enter 2024-03-25T18:14:20.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 10 librados: watch_flush exit 2024-03-25T18:14:20.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 20 client.4195.objecter shutdown clearing up homeless session... 2024-03-25T18:14:20.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 10 client.4195.objecter successfully canceled tick 2024-03-25T18:14:20.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 >> v1:172.21.15.191:6800/1552703795 conn(0x7fb81006e0f0 legacy=0x7fb8100705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:20.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 10 monclient: shutdown 2024-03-25T18:14:20.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:20.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 >> v1:172.21.15.191:6789/0 conn(0x7fb8300a98f0 legacy=0x7fb830137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:20.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 shutdown_connections 2024-03-25T18:14:20.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 >> 172.21.15.162:0/4113090576 conn(0x7fb8300b7ea0 msgr2=0x7fb83013c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:20.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 shutdown_connections 2024-03-25T18:14:20.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 1 -- 172.21.15.162:0/4113090576 wait complete. 2024-03-25T18:14:20.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:20.240+0000 7fb83482a640 1 librados: shutdown 2024-03-25T18:14:20.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:20.251 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:10.057 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.282 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:10.071 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.306 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:10.101 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.404 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:10.212 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.428 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:10.232 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.440 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:10.235 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.463 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:10.273 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.515 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:10.322 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.535 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:10.347 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.560 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:10.349 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.575 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:10.370 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.707 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:10.502 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.727 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:10.531 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.758 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:10.569 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.826 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:10.638 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.830 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:10.619 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.846 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:10.641 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.877 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:10.685 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.931 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:10.739 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.975 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:10.770 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:20.994 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:10.801 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.048 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:10.856 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.053 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:10.864 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.242 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:11.037 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.259 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:11.063 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.302 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:11.091 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.313 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:11.117 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.349 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:11.143 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.356 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:11.145 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.371 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:11.184 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.403 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:11.197 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.426 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:11.238 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.601 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:11.412 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.657 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:11.467 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.781 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:11.575 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:21.869 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:11.664 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.453 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:16.242 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.499 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:16.307 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.511 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:16.323 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.531 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:16.335 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.617 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:16.424 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.684 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:16.479 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.057 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:27.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.199+0000 7fd2c987d640 10 monclient: get_monmap_and_config 2024-03-25T18:14:27.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.199+0000 7fd2c987d640 10 monclient: build_initial_monmap 2024-03-25T18:14:27.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.199+0000 7fd2c987d640 10 monclient: monmap: 2024-03-25T18:14:27.200 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:27.200 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:27.201 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:14:27.200381+0000 2024-03-25T18:14:27.201 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:14:27.200381+0000 2024-03-25T18:14:27.201 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:27.201 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:27.201 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:27.201 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:27.201 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:27.201 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.200+0000 7fd2c987d640 1 Processor -- start 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.200+0000 7fd2c987d640 1 -- start start 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.200+0000 7fd2c987d640 10 monclient: init 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 10 monclient(hunting): picked mon.noname-c con 0x7fd2c40b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 10 monclient(hunting): picked mon.noname-a con 0x7fd2c40b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2c414be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40af940 con 0x7fd2c40b1730 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40b0a70 con 0x7fd2c40b4d60 2024-03-25T18:14:27.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40b0d50 con 0x7fd2c414be90 2024-03-25T18:14:27.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:27.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c987d640 10 monclient(hunting): authenticate will time out at 7566.475153s 2024-03-25T18:14:27.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c27fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fd2c40b1730 0x7fd2c40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:57124/0 (socket says 172.21.15.162:57124) 2024-03-25T18:14:27.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.201+0000 7fd2c27fc640 1 -- 172.21.15.162:0/598306592 learned_addr learned my addr 172.21.15.162:0/598306592 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:27.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098897533 0 0) 0x7fd2c40b0a70 con 0x7fd2c40b4d60 2024-03-25T18:14:27.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 10 monclient(hunting): my global_id is 4198 2024-03-25T18:14:27.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac003520 con 0x7fd2c40b4d60 2024-03-25T18:14:27.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578176880 0 0) 0x7fd2c40af940 con 0x7fd2c40b1730 2024-03-25T18:14:27.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:14:27.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.205+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c40b0a70 con 0x7fd2c40b1730 2024-03-25T18:14:27.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.206+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952356888 0 0) 0x7fd2c40b0d50 con 0x7fd2c414be90 2024-03-25T18:14:27.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.206+0000 7fd2c1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.206+0000 7fd2c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.206+0000 7fd2c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.206+0000 7fd2c1ffb640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:14:27.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.206+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c40af940 con 0x7fd2c414be90 2024-03-25T18:14:27.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095112757 0 0) 0x7fd2ac003520 con 0x7fd2c40b4d60 2024-03-25T18:14:27.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 >> v1:172.21.15.162:6790/0 conn(0x7fd2c414be90 legacy=0x7fd2c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 >> v1:172.21.15.162:6789/0 conn(0x7fd2c40b1730 legacy=0x7fd2c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40b1030 con 0x7fd2c40b4d60 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd2b4001690 con 0x7fd2c40b4d60 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c987d640 5 monclient: authenticate success, global_id 4198 2024-03-25T18:14:27.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c1ffb640 10 monclient: dump: 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:27.212 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:27.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.209+0000 7fd2c987d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:27.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.211+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b40017f0 con 0x7fd2c40b4d60 2024-03-25T18:14:27.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.211+0000 7fd2c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:27.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.211+0000 7fd2c987d640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:27.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.211+0000 7fd2c987d640 4 set_mon_vals no callback set 2024-03-25T18:14:27.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c1ffb640 1 -- 172.21.15.162:0/598306592 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd2b4001c00 con 0x7fd2c40b4d60 2024-03-25T18:14:27.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c1ffb640 10 monclient: dump: 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:27.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c987d640 10 monclient: shutdown 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c987d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c987d640 1 -- 172.21.15.162:0/598306592 >> v1:172.21.15.191:6789/0 conn(0x7fd2c40b4d60 legacy=0x7fd2c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c987d640 1 -- 172.21.15.162:0/598306592 shutdown_connections 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c987d640 1 -- 172.21.15.162:0/598306592 >> 172.21.15.162:0/598306592 conn(0x7fd2c40617a0 msgr2=0x7fd2c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c987d640 1 -- 172.21.15.162:0/598306592 shutdown_connections 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.212+0000 7fd2c987d640 1 -- 172.21.15.162:0/598306592 wait complete. 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.213+0000 7fd2c987d640 10 monclient: build_initial_monmap 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.213+0000 7fd2c987d640 10 monclient: monmap: 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:14:27.216 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:27.217 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:27.217 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:27.217 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:27.217 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:27.217 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:27.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.214+0000 7fd2c987d640 1 librados: starting msgr at 2024-03-25T18:14:27.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.214+0000 7fd2c987d640 1 librados: starting objecter 2024-03-25T18:14:27.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.214+0000 7fd2c987d640 1 Processor -- start 2024-03-25T18:14:27.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.214+0000 7fd2c987d640 1 -- start start 2024-03-25T18:14:27.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.214+0000 7fd2c987d640 1 librados: setting wanted keys 2024-03-25T18:14:27.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.214+0000 7fd2c987d640 1 librados: calling monclient init 2024-03-25T18:14:27.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.214+0000 7fd2c987d640 10 monclient: init 2024-03-25T18:14:27.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 10 monclient(hunting): picked mon.noname-a con 0x7fd2c40b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 10 monclient(hunting): picked mon.noname-c con 0x7fd2c40b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 10 monclient(hunting): picked mon.noname-b con 0x7fd2c414be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4140a80 con 0x7fd2c40b1730 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4141bc0 con 0x7fd2c414be90 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4141ea0 con 0x7fd2c40b4d60 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c987d640 10 monclient(hunting): authenticate will time out at 7566.489211s 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c2ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fd2c40b1730 0x7fd2c4135200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:57140/0 (socket says 172.21.15.162:57140) 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.215+0000 7fd2c2ffd640 1 -- 172.21.15.162:0/3572379998 learned_addr learned my addr 172.21.15.162:0/3572379998 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.216+0000 7fd2abfff640 10 client.?.objecter ms_handle_connect 0x7fd2c414be90 2024-03-25T18:14:27.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.216+0000 7fd2abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.216+0000 7fd2abfff640 10 client.?.objecter ms_handle_connect 0x7fd2c40b1730 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.216+0000 7fd2abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.216+0000 7fd2abfff640 10 client.?.objecter ms_handle_connect 0x7fd2c40b4d60 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.216+0000 7fd2abfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784628279 0 0) 0x7fd2c4141ea0 con 0x7fd2c40b4d60 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b0004eb0 con 0x7fd2c40b4d60 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830846183 0 0) 0x7fd2c4141bc0 con 0x7fd2c414be90 2024-03-25T18:14:27.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.218+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4141ea0 con 0x7fd2c414be90 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.219+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614070641 0 0) 0x7fd2c40b1030 con 0x7fd2c40b1730 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.219+0000 7fd2abfff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.219+0000 7fd2abfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.219+0000 7fd2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.219+0000 7fd2abfff640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:14:27.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.219+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4141bc0 con 0x7fd2c40b1730 2024-03-25T18:14:27.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.224+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2572612975 0 0) 0x7fd2c4141ea0 con 0x7fd2c414be90 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.224+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c40b1030 con 0x7fd2c414be90 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.224+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd2bc001b50 con 0x7fd2c414be90 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.224+0000 7fd2abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.224+0000 7fd2abfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.224+0000 7fd2abfff640 10 monclient(hunting): dump: 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.226 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:27.227 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:27.227 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:27.227 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:27.227 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:27.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.224+0000 7fd2abfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:27.227 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:17.021 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.225+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2296918567 0 0) 0x7fd2b0004eb0 con 0x7fd2c40b4d60 2024-03-25T18:14:27.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.225+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c4141ea0 con 0x7fd2c40b4d60 2024-03-25T18:14:27.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.226+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd2b8001970 con 0x7fd2c40b4d60 2024-03-25T18:14:27.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.226+0000 7fd2abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.226+0000 7fd2abfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.226+0000 7fd2abfff640 10 monclient(hunting): dump: 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:27.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.226+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 645186653 0 0) 0x7fd2c4141bc0 con 0x7fd2c40b1730 2024-03-25T18:14:27.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.226+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2b0004eb0 con 0x7fd2c40b1730 2024-03-25T18:14:27.233 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3535482701 0 0) 0x7fd2c40b1030 con 0x7fd2c414be90 2024-03-25T18:14:27.233 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 >> v1:172.21.15.162:6790/0 conn(0x7fd2c40b4d60 legacy=0x7fd2c41d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.233 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 >> v1:172.21.15.162:6789/0 conn(0x7fd2c40b1730 legacy=0x7fd2c4135200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 1 monclient: found mon.b 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c4142180 con 0x7fd2c414be90 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 10 monclient: _finish_auth 0 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2c987d640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2c987d640 10 monclient: _renew_subs 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2c987d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:27.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c41425a0 con 0x7fd2c414be90 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2c987d640 10 client.4201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2c987d640 10 monclient: _renew_subs 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.229+0000 7fd2c987d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.230+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c4140710 con 0x7fd2c414be90 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.230+0000 7fd2c987d640 1 librados: init done 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2c987d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2c987d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2c987d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2bc001fe0 con 0x7fd2c414be90 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c40627b0 con 0x7fd2c414be90 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd2bc003b30 con 0x7fd2c414be90 2024-03-25T18:14:27.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2abfff640 10 monclient: dump: 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:27.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.231+0000 7fd2c17fa640 4 set_mon_vals no callback set 2024-03-25T18:14:27.258 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:17.068 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.339 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.338+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fd2bc019b00 con 0x7fd2c414be90 2024-03-25T18:14:27.340 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.339+0000 7fd2c2ffd640 10 monclient: get_auth_request con 0x7fd2b006e200 auth_method 0 2024-03-25T18:14:27.340 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.339+0000 7fd2abfff640 10 client.4201.objecter ms_handle_connect 0x7fd2b006e200 2024-03-25T18:14:27.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.590+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fd2bc053780 con 0x7fd2c414be90 2024-03-25T18:14:27.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.590+0000 7fd2abfff640 10 client.4201.objecter ms_dispatch 0x7fd2c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:27.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.590+0000 7fd2abfff640 3 client.4201.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.590+0000 7fd2abfff640 3 client.4201.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.590+0000 7fd2abfff640 20 client.4201.objecter dump_active .. 0 homeless 2024-03-25T18:14:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.590+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2bc04f0c0 con 0x7fd2c414be90 2024-03-25T18:14:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.590+0000 7fd2abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.590+0000 7fd2abfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:27.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.893+0000 7fd2c987d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:27.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.893+0000 7fd2c987d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:27.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.893+0000 7fd2c987d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:27.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.893+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2c41d78e0 con 0x7fd2c414be90 2024-03-25T18:14:27.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.919+0000 7fd2abfff640 1 -- 172.21.15.162:0/3572379998 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fd2bc0511c0 con 0x7fd2c414be90 2024-03-25T18:14:27.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.919+0000 7fd2abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:27.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.919+0000 7fd2abfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:27.920 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:14:27.920 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:27.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 10 librados: watch_flush enter 2024-03-25T18:14:27.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 10 librados: watch_flush exit 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 20 client.4201.objecter shutdown clearing up homeless session... 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 10 client.4201.objecter successfully canceled tick 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 >> v1:172.21.15.191:6800/1552703795 conn(0x7fd2b006e200 legacy=0x7fd2b00706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 10 monclient: shutdown 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 >> v1:172.21.15.191:6789/0 conn(0x7fd2c414be90 legacy=0x7fd2c41e3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 shutdown_connections 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 >> 172.21.15.162:0/3572379998 conn(0x7fd2c40617a0 msgr2=0x7fd2c413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 shutdown_connections 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 1 -- 172.21.15.162:0/3572379998 wait complete. 2024-03-25T18:14:27.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:27.921+0000 7fd2c987d640 1 librados: shutdown 2024-03-25T18:14:27.933 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:31.834 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:21.628 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:31.838 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:21.633 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.048 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:21.837 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.054 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:21.843 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.207 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:22.014 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.212 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:22.019 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.279 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:22.091 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.283 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:22.095 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.346 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:22.149 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.351 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:22.155 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.689 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:22.497 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.693 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:22.501 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.770 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:22.565 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.776 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:22.571 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:33.214 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:23.024 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:33.218 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:23.028 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:34.739 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:34.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.827+0000 7ffb7354b640 10 monclient: get_monmap_and_config 2024-03-25T18:14:34.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.827+0000 7ffb7354b640 10 monclient: build_initial_monmap 2024-03-25T18:14:34.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.827+0000 7ffb7354b640 10 monclient: monmap: 2024-03-25T18:14:34.828 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:34.828 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:34.829 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:14:34.828650+0000 2024-03-25T18:14:34.829 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:14:34.828650+0000 2024-03-25T18:14:34.829 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:34.829 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:34.829 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:34.829 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:34.829 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:34.829 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:34.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.828+0000 7ffb7354b640 1 Processor -- start 2024-03-25T18:14:34.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.828+0000 7ffb7354b640 1 -- start start 2024-03-25T18:14:34.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.828+0000 7ffb7354b640 10 monclient: init 2024-03-25T18:14:34.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:34.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:34.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 10 monclient(hunting): picked mon.noname-a con 0x7ffb6c0b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:34.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6c0b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:34.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 10 monclient(hunting): picked mon.noname-c con 0x7ffb6c14be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:34.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c0af940 con 0x7ffb6c0b4d60 2024-03-25T18:14:34.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c0b0a70 con 0x7ffb6c14be90 2024-03-25T18:14:34.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c0b0d50 con 0x7ffb6c0b1730 2024-03-25T18:14:34.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:34.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb7354b640 10 monclient(hunting): authenticate will time out at 7574.102940s 2024-03-25T18:14:34.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb70ae6640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7ffb6c0b1730 0x7ffb6c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:48138/0 (socket says 172.21.15.162:48138) 2024-03-25T18:14:34.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.829+0000 7ffb70ae6640 1 -- 172.21.15.162:0/3181676245 learned_addr learned my addr 172.21.15.162:0/3181676245 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:34.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823681383 0 0) 0x7ffb6c0b0d50 con 0x7ffb6c0b1730 2024-03-25T18:14:34.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:34.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:34.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:34.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb58003520 con 0x7ffb6c0b1730 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386010530 0 0) 0x7ffb6c0b0a70 con 0x7ffb6c14be90 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c0b0d50 con 0x7ffb6c14be90 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771437495 0 0) 0x7ffb6c0af940 con 0x7ffb6c0b4d60 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:14:34.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.833+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c0b0a70 con 0x7ffb6c0b4d60 2024-03-25T18:14:34.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.836+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 663042206 0 0) 0x7ffb58003520 con 0x7ffb6c0b1730 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.836+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 >> v1:172.21.15.191:6789/0 conn(0x7ffb6c14be90 legacy=0x7ffb6c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.836+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 >> v1:172.21.15.162:6789/0 conn(0x7ffb6c0b4d60 legacy=0x7ffb6c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 1 monclient: found mon.noname-b 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c0b1030 con 0x7ffb6c0b1730 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffb5c001690 con 0x7ffb6c0b1730 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:34.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb63fff640 10 monclient: dump: 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb7354b640 5 monclient: authenticate success, global_id 4214 2024-03-25T18:14:34.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.837+0000 7ffb7354b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:34.841 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.839+0000 7ffb63fff640 1 -- 172.21.15.162:0/3181676245 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb5c0017f0 con 0x7ffb6c0b1730 2024-03-25T18:14:34.841 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.839+0000 7ffb63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:34.841 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.839+0000 7ffb7354b640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:34.841 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.839+0000 7ffb7354b640 4 set_mon_vals no callback set 2024-03-25T18:14:34.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.840+0000 7ffb7354b640 10 monclient: shutdown 2024-03-25T18:14:34.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.840+0000 7ffb7354b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:34.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.840+0000 7ffb7354b640 1 -- 172.21.15.162:0/3181676245 >> v1:172.21.15.162:6790/0 conn(0x7ffb6c0b1730 legacy=0x7ffb6c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:34.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.840+0000 7ffb7354b640 1 -- 172.21.15.162:0/3181676245 shutdown_connections 2024-03-25T18:14:34.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.840+0000 7ffb7354b640 1 -- 172.21.15.162:0/3181676245 >> 172.21.15.162:0/3181676245 conn(0x7ffb6c0617a0 msgr2=0x7ffb6c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:34.842 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.841+0000 7ffb7354b640 1 -- 172.21.15.162:0/3181676245 shutdown_connections 2024-03-25T18:14:34.843 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.841+0000 7ffb7354b640 1 -- 172.21.15.162:0/3181676245 wait complete. 2024-03-25T18:14:34.843 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.842+0000 7ffb7354b640 10 monclient: build_initial_monmap 2024-03-25T18:14:34.843 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.842+0000 7ffb7354b640 10 monclient: monmap: 2024-03-25T18:14:34.843 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:34.843 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:34.843 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:34.844 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:14:34.844 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:34.844 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:34.844 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:34.844 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:34.844 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:34.844 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.843+0000 7ffb7354b640 1 librados: starting msgr at 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.843+0000 7ffb7354b640 1 librados: starting objecter 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.843+0000 7ffb7354b640 1 Processor -- start 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.843+0000 7ffb7354b640 1 -- start start 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.844+0000 7ffb7354b640 1 librados: setting wanted keys 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.844+0000 7ffb7354b640 1 librados: calling monclient init 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.844+0000 7ffb7354b640 10 monclient: init 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.844+0000 7ffb7354b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:34.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.844+0000 7ffb7354b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:34.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.844+0000 7ffb7354b640 10 monclient(hunting): picked mon.noname-a con 0x7ffb6c0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:34.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.844+0000 7ffb7354b640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6c0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:34.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.844+0000 7ffb7354b640 10 monclient(hunting): picked mon.noname-c con 0x7ffb6c14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:34.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.845+0000 7ffb7354b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c1d78c0 con 0x7ffb6c0b1730 2024-03-25T18:14:34.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.845+0000 7ffb7354b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c1d7ba0 con 0x7ffb6c0b4d60 2024-03-25T18:14:34.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.845+0000 7ffb7354b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6c1d8740 con 0x7ffb6c14be90 2024-03-25T18:14:34.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.845+0000 7ffb7354b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:34.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.845+0000 7ffb7354b640 10 monclient(hunting): authenticate will time out at 7574.118670s 2024-03-25T18:14:34.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.845+0000 7ffb70ae6640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ffb6c0b4d60 0x7ffb6c13e5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:59334/0 (socket says 172.21.15.162:59334) 2024-03-25T18:14:34.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.845+0000 7ffb70ae6640 1 -- 172.21.15.162:0/2531165748 learned_addr learned my addr 172.21.15.162:0/2531165748 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:34.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.846+0000 7ffb61ffb640 10 client.?.objecter ms_handle_connect 0x7ffb6c14be90 2024-03-25T18:14:34.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.846+0000 7ffb61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:34.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.846+0000 7ffb61ffb640 10 client.?.objecter ms_handle_connect 0x7ffb6c0b4d60 2024-03-25T18:14:34.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.846+0000 7ffb61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:34.849 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.848+0000 7ffb61ffb640 10 client.?.objecter ms_handle_connect 0x7ffb6c0b1730 2024-03-25T18:14:34.849 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.848+0000 7ffb61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:34.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375712702 0 0) 0x7ffb6c1d7ba0 con 0x7ffb6c0b4d60 2024-03-25T18:14:34.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:34.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:34.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:34.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:14:34.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb50004eb0 con 0x7ffb6c0b4d60 2024-03-25T18:14:34.851 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670165468 0 0) 0x7ffb6c1d8740 con 0x7ffb6c14be90 2024-03-25T18:14:34.851 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:34.851 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:34.851 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:34.851 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:14:34.851 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.849+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c1d7ba0 con 0x7ffb6c14be90 2024-03-25T18:14:34.851 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.850+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863520833 0 0) 0x7ffb6c1d78c0 con 0x7ffb6c0b1730 2024-03-25T18:14:34.851 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.850+0000 7ffb61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:34.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.850+0000 7ffb61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:34.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.850+0000 7ffb61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:34.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.850+0000 7ffb61ffb640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:14:34.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.850+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c1d8740 con 0x7ffb6c0b1730 2024-03-25T18:14:34.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2607505238 0 0) 0x7ffb50004eb0 con 0x7ffb6c0b4d60 2024-03-25T18:14:34.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb50005620 con 0x7ffb6c0b4d60 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1087798492 0 0) 0x7ffb6c1d8740 con 0x7ffb6c0b1730 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb50004eb0 con 0x7ffb6c0b1730 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffb68001870 con 0x7ffb6c0b1730 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:34.857 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.855+0000 7ffb61ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.857+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1982813392 0 0) 0x7ffb6c1d7ba0 con 0x7ffb6c14be90 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.857+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb50006150 con 0x7ffb6c14be90 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.857+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffb64001b50 con 0x7ffb6c14be90 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.857+0000 7ffb61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.857+0000 7ffb61ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.857+0000 7ffb61ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:34.858 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:34.859 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:34.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1100509429 0 0) 0x7ffb50004eb0 con 0x7ffb6c0b1730 2024-03-25T18:14:34.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 >> v1:172.21.15.162:6790/0 conn(0x7ffb6c14be90 legacy=0x7ffb6c1e21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:34.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 >> v1:172.21.15.191:6789/0 conn(0x7ffb6c0b4d60 legacy=0x7ffb6c13e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:34.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 1 monclient: found mon.a 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c1458b0 con 0x7ffb6c0b1730 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.859+0000 7ffb61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 10 monclient: _renew_subs 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c13ece0 con 0x7ffb6c0b1730 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 10 monclient: _renew_subs 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:34.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c13e060 con 0x7ffb6c0b1730 2024-03-25T18:14:34.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.860+0000 7ffb7354b640 1 librados: init done 2024-03-25T18:14:34.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.861+0000 7ffb7354b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:34.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.861+0000 7ffb7354b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:34.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.861+0000 7ffb7354b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:34.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:34.861+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6c13e060 con 0x7ffb6c0b1730 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.110+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb68001d60 con 0x7ffb6c0b1730 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.110+0000 7ffb61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.110+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffb68001870 con 0x7ffb6c0b1730 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.110+0000 7ffb61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.110+0000 7ffb61ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.110+0000 7ffb61ffb640 10 monclient: dump: 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.110+0000 7ffb637fe640 4 set_mon_vals no callback set 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.111+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7ffb680194e0 con 0x7ffb6c0b1730 2024-03-25T18:14:35.114 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.112+0000 7ffb70ae6640 10 monclient: get_auth_request con 0x7ffb5006e820 auth_method 0 2024-03-25T18:14:35.114 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.113+0000 7ffb61ffb640 10 client.4221.objecter ms_handle_connect 0x7ffb5006e820 2024-03-25T18:14:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.212+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7ffb68053120 con 0x7ffb6c0b1730 2024-03-25T18:14:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.212+0000 7ffb61ffb640 10 client.4221.objecter ms_dispatch 0x7ffb6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.212+0000 7ffb61ffb640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.212+0000 7ffb61ffb640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.212+0000 7ffb61ffb640 20 client.4221.objecter dump_active .. 0 homeless 2024-03-25T18:14:35.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.252+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb6804ea60 con 0x7ffb6c0b1730 2024-03-25T18:14:35.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.253+0000 7ffb61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:35.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.253+0000 7ffb61ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:35.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.431+0000 7ffb7354b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:35.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.431+0000 7ffb7354b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:35.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.431+0000 7ffb7354b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:14:35.433 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.431+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb6c13e060 con 0x7ffb6c0b1730 2024-03-25T18:14:35.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.474+0000 7ffb61ffb640 1 -- 172.21.15.162:0/2531165748 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7ffb68050b60 con 0x7ffb6c0b1730 2024-03-25T18:14:35.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.474+0000 7ffb61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:35.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.474+0000 7ffb61ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:35.476 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:14:35.476 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:35.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 10 librados: watch_flush enter 2024-03-25T18:14:35.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 10 librados: watch_flush exit 2024-03-25T18:14:35.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 20 client.4221.objecter shutdown clearing up homeless session... 2024-03-25T18:14:35.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 10 client.4221.objecter successfully canceled tick 2024-03-25T18:14:35.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 >> v1:172.21.15.191:6800/1552703795 conn(0x7ffb5006e820 legacy=0x7ffb50070ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 10 monclient: shutdown 2024-03-25T18:14:35.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:35.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 >> v1:172.21.15.162:6789/0 conn(0x7ffb6c0b1730 legacy=0x7ffb6c1d8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 shutdown_connections 2024-03-25T18:14:35.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.477+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 >> 172.21.15.162:0/2531165748 conn(0x7ffb6c0617a0 msgr2=0x7ffb6c0b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:35.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.478+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 shutdown_connections 2024-03-25T18:14:35.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.478+0000 7ffb7354b640 1 -- 172.21.15.162:0/2531165748 wait complete. 2024-03-25T18:14:35.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:35.478+0000 7ffb7354b640 1 librados: shutdown 2024-03-25T18:14:35.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:36.006 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:00:25.801 35336== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:36.271 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:00:26.075 35465== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:36.437 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:00:26.244 35464== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:36.476 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:00:26.288 35334== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:36.663 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:00:26.452 35337== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:36.849 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:00:26.657 35463== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:37.417 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:00:27.212 35335== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:37.556 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:00:27.366 35462== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:42.294 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.389+0000 7ff68dea8640 10 monclient: get_monmap_and_config 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.389+0000 7ff68dea8640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.389+0000 7ff68dea8640 10 monclient: monmap: 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:14:42.390090+0000 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:14:42.390090+0000 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:42.390 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.389+0000 7ff68dea8640 1 Processor -- start 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.389+0000 7ff68dea8640 1 -- start start 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.389+0000 7ff68dea8640 10 monclient: init 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 10 monclient(hunting): picked mon.noname-a con 0x7ff6880b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 10 monclient(hunting): picked mon.noname-c con 0x7ff6880b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 10 monclient(hunting): picked mon.noname-b con 0x7ff68814be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:42.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6880af940 con 0x7ff6880b4d60 2024-03-25T18:14:42.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6880b0a70 con 0x7ff6880b1730 2024-03-25T18:14:42.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6880b0d50 con 0x7ff68814be90 2024-03-25T18:14:42.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.390+0000 7ff68dea8640 10 monclient(hunting): authenticate will time out at 7581.664439s 2024-03-25T18:14:42.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.391+0000 7ff6877fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ff6880b4d60 0x7ff68813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:37246/0 (socket says 172.21.15.162:37246) 2024-03-25T18:14:42.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.391+0000 7ff687fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7ff68814be90 0x7ff68814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:48156/0 (socket says 172.21.15.162:48156) 2024-03-25T18:14:42.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.391+0000 7ff687fff640 1 -- 172.21.15.162:0/3080876518 learned_addr learned my addr 172.21.15.162:0/3080876518 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:42.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327230230 0 0) 0x7ff6880b0d50 con 0x7ff68814be90 2024-03-25T18:14:42.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:14:42.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff670003520 con 0x7ff68814be90 2024-03-25T18:14:42.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959100573 0 0) 0x7ff6880af940 con 0x7ff6880b4d60 2024-03-25T18:14:42.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:14:42.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.394+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6880b0d50 con 0x7ff6880b4d60 2024-03-25T18:14:42.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.395+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362574326 0 0) 0x7ff6880b0a70 con 0x7ff6880b1730 2024-03-25T18:14:42.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.395+0000 7ff6867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.395+0000 7ff6867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.395+0000 7ff6867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.395+0000 7ff6867fc640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:14:42.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.395+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6700039d0 con 0x7ff6880b1730 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2974531099 0 0) 0x7ff670003520 con 0x7ff68814be90 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 >> v1:172.21.15.191:6789/0 conn(0x7ff6880b1730 legacy=0x7ff6880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 >> v1:172.21.15.162:6789/0 conn(0x7ff6880b4d60 legacy=0x7ff68813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 1 monclient: found mon.noname-b 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6880b1030 con 0x7ff68814be90 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff680001a10 con 0x7ff68814be90 2024-03-25T18:14:42.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff6867fc640 10 monclient: dump: 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff68dea8640 5 monclient: authenticate success, global_id 4220 2024-03-25T18:14:42.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.399+0000 7ff68dea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.401+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff680001b70 con 0x7ff68814be90 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.401+0000 7ff6867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.401+0000 7ff6867fc640 1 -- 172.21.15.162:0/3080876518 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff680001f80 con 0x7ff68814be90 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.401+0000 7ff6867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.401+0000 7ff6867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.401+0000 7ff6867fc640 10 monclient: dump: 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.404 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.401+0000 7ff68dea8640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.401+0000 7ff68dea8640 4 set_mon_vals no callback set 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.402+0000 7ff68dea8640 10 monclient: shutdown 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.402+0000 7ff68dea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.402+0000 7ff68dea8640 1 -- 172.21.15.162:0/3080876518 >> v1:172.21.15.162:6790/0 conn(0x7ff68814be90 legacy=0x7ff68814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.402+0000 7ff68dea8640 1 -- 172.21.15.162:0/3080876518 shutdown_connections 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.402+0000 7ff68dea8640 1 -- 172.21.15.162:0/3080876518 >> 172.21.15.162:0/3080876518 conn(0x7ff6880617a0 msgr2=0x7ff688061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.402+0000 7ff68dea8640 1 -- 172.21.15.162:0/3080876518 shutdown_connections 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.402+0000 7ff68dea8640 1 -- 172.21.15.162:0/3080876518 wait complete. 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.403+0000 7ff68dea8640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.403+0000 7ff68dea8640 10 monclient: monmap: 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:42.406 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:42.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.403+0000 7ff68dea8640 1 librados: starting msgr at 2024-03-25T18:14:42.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.403+0000 7ff68dea8640 1 librados: starting objecter 2024-03-25T18:14:42.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 1 Processor -- start 2024-03-25T18:14:42.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 1 -- start start 2024-03-25T18:14:42.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 1 librados: setting wanted keys 2024-03-25T18:14:42.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 1 librados: calling monclient init 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 10 monclient: init 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 10 monclient(hunting): picked mon.noname-c con 0x7ff6880b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 10 monclient(hunting): picked mon.noname-b con 0x7ff6880b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 10 monclient(hunting): picked mon.noname-a con 0x7ff68814be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff68813afc0 con 0x7ff68814be90 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff688134f90 con 0x7ff6880b4d60 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.404+0000 7ff68dea8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6881411c0 con 0x7ff6880b1730 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.405+0000 7ff68dea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.405+0000 7ff68dea8640 10 monclient(hunting): authenticate will time out at 7581.678653s 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.405+0000 7ff6877fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7ff6880b1730 0x7ff68813b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:48170/0 (socket says 172.21.15.162:48170) 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.405+0000 7ff6877fe640 1 -- 172.21.15.162:0/3778897181 learned_addr learned my addr 172.21.15.162:0/3778897181 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.405+0000 7ff66ffff640 10 client.?.objecter ms_handle_connect 0x7ff6880b1730 2024-03-25T18:14:42.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.405+0000 7ff66ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.406+0000 7ff66ffff640 10 client.?.objecter ms_handle_connect 0x7ff68814be90 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.406+0000 7ff66ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.406+0000 7ff66ffff640 10 client.?.objecter ms_handle_connect 0x7ff6880b4d60 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.406+0000 7ff66ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644045631 0 0) 0x7ff68813afc0 con 0x7ff68814be90 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 10 monclient(hunting): my global_id is 4227 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff674004eb0 con 0x7ff68814be90 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330391689 0 0) 0x7ff6881411c0 con 0x7ff6880b1730 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:14:42.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.408+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff68813afc0 con 0x7ff6880b1730 2024-03-25T18:14:42.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.409+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931897911 0 0) 0x7ff688134f90 con 0x7ff6880b4d60 2024-03-25T18:14:42.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.409+0000 7ff66ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.409+0000 7ff66ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.409+0000 7ff66ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.409+0000 7ff66ffff640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:14:42.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.409+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6881411c0 con 0x7ff6880b4d60 2024-03-25T18:14:42.414 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.414+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163603628 0 0) 0x7ff674004eb0 con 0x7ff68814be90 2024-03-25T18:14:42.415 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.414+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff688134f90 con 0x7ff68814be90 2024-03-25T18:14:42.416 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.415+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4201861989 0 0) 0x7ff68813afc0 con 0x7ff6880b1730 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.415+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff674004eb0 con 0x7ff6880b1730 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.415+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff6780019f0 con 0x7ff6880b1730 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.415+0000 7ff66ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.415+0000 7ff66ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.415+0000 7ff66ffff640 10 monclient(hunting): dump: 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:42.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.415+0000 7ff66ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:42.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.416+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3029029411 0 0) 0x7ff6881411c0 con 0x7ff6880b4d60 2024-03-25T18:14:42.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.416+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff68813afc0 con 0x7ff6880b4d60 2024-03-25T18:14:42.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.417+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff67c0018b0 con 0x7ff6880b4d60 2024-03-25T18:14:42.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.417+0000 7ff66ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.417+0000 7ff66ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:42.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.417+0000 7ff66ffff640 10 monclient(hunting): dump: 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:42.419 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff680001e80 con 0x7ff68814be90 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 10 monclient(hunting): dump: 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:42.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3818852380 0 0) 0x7ff674004eb0 con 0x7ff6880b1730 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 >> v1:172.21.15.191:6789/0 conn(0x7ff6880b4d60 legacy=0x7ff688134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 >> v1:172.21.15.162:6789/0 conn(0x7ff68814be90 legacy=0x7ff688134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 1 monclient: found mon.c 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6881414a0 con 0x7ff6880b1730 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff66ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 5 monclient: authenticate success, global_id 4223 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 10 monclient: _renew_subs 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:42.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68813bb60 con 0x7ff6880b1730 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 10 client.4223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 10 monclient: _renew_subs 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff688141aa0 con 0x7ff6880b1730 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.420+0000 7ff68dea8640 1 librados: init done 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.421+0000 7ff66dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.422+0000 7ff66dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.422+0000 7ff66dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:42.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.422+0000 7ff66dffb640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff668005080 con 0x7ff6880b1730 2024-03-25T18:14:42.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.739+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6780047e0 con 0x7ff6880b1730 2024-03-25T18:14:42.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.739+0000 7ff66ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.739+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff678004af0 con 0x7ff6880b1730 2024-03-25T18:14:42.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.739+0000 7ff66ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.739+0000 7ff66ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.739+0000 7ff66ffff640 10 monclient: dump: 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.739+0000 7ff685ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:42.742 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.740+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7ff67801aac0 con 0x7ff6880b1730 2024-03-25T18:14:42.743 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.741+0000 7ff686ffd640 10 monclient: get_auth_request con 0x7ff67406e110 auth_method 0 2024-03-25T18:14:42.743 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.741+0000 7ff66ffff640 10 client.4223.objecter ms_handle_connect 0x7ff67406e110 2024-03-25T18:14:42.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.745+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7ff6780547b0 con 0x7ff6880b1730 2024-03-25T18:14:42.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.745+0000 7ff66ffff640 10 client.4223.objecter ms_dispatch 0x7ff688000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:42.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.745+0000 7ff66ffff640 3 client.4223.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:42.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.745+0000 7ff66ffff640 3 client.4223.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:42.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.745+0000 7ff66ffff640 20 client.4223.objecter dump_active .. 0 homeless 2024-03-25T18:14:42.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.745+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6780500f0 con 0x7ff6880b1730 2024-03-25T18:14:42.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.745+0000 7ff66ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:42.745+0000 7ff66ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:43.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.097+0000 7ff66dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:43.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.097+0000 7ff66dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:43.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.097+0000 7ff66dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:14:43.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.097+0000 7ff66dffb640 1 -- 172.21.15.162:0/3778897181 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff668005330 con 0x7ff6880b1730 2024-03-25T18:14:43.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.169+0000 7ff66ffff640 1 -- 172.21.15.162:0/3778897181 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7ff6780521f0 con 0x7ff6880b1730 2024-03-25T18:14:43.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.169+0000 7ff66ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:43.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.169+0000 7ff66ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:43.170 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:14:43.171 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:43.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 10 librados: watch_flush enter 2024-03-25T18:14:43.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 10 librados: watch_flush exit 2024-03-25T18:14:43.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 20 client.4223.objecter shutdown clearing up homeless session... 2024-03-25T18:14:43.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 10 client.4223.objecter successfully canceled tick 2024-03-25T18:14:43.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 1 -- 172.21.15.162:0/3778897181 >> v1:172.21.15.191:6800/1552703795 conn(0x7ff67406e110 legacy=0x7ff6740705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 10 monclient: shutdown 2024-03-25T18:14:43.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:43.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 1 -- 172.21.15.162:0/3778897181 >> v1:172.21.15.162:6790/0 conn(0x7ff6880b1730 legacy=0x7ff68813b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 1 -- 172.21.15.162:0/3778897181 shutdown_connections 2024-03-25T18:14:43.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.172+0000 7ff66dffb640 1 -- 172.21.15.162:0/3778897181 >> 172.21.15.162:0/3778897181 conn(0x7ff6880617a0 msgr2=0x7ff6880b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:43.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.173+0000 7ff66dffb640 1 -- 172.21.15.162:0/3778897181 shutdown_connections 2024-03-25T18:14:43.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.173+0000 7ff66dffb640 1 -- 172.21.15.162:0/3778897181 wait complete. 2024-03-25T18:14:43.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:43.173+0000 7ff66dffb640 1 librados: shutdown 2024-03-25T18:14:43.186 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:44.778 INFO:tasks.ceph.osd.7.smithi191.stderr:2024-03-25T18:14:44.774+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:14:45.039 INFO:tasks.ceph.osd.6.smithi191.stderr:2024-03-25T18:14:45.035+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:14:45.204 INFO:tasks.ceph.osd.2.smithi162.stderr:2024-03-25T18:14:45.200+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:14:45.219 INFO:tasks.ceph.osd.0.smithi162.stderr:2024-03-25T18:14:45.216+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:14:45.245 INFO:tasks.ceph.osd.5.smithi191.stderr:2024-03-25T18:14:45.242+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:14:45.586 INFO:tasks.ceph.osd.4.smithi191.stderr:2024-03-25T18:14:45.582+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:14:45.978 INFO:tasks.ceph.osd.3.smithi162.stderr:2024-03-25T18:14:45.974+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:14:46.496 INFO:tasks.ceph.osd.1.smithi162.stderr:2024-03-25T18:14:46.491+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:14:49.993 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:50.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.106+0000 7fe36c790640 10 monclient: get_monmap_and_config 2024-03-25T18:14:50.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.106+0000 7fe36c790640 10 monclient: build_initial_monmap 2024-03-25T18:14:50.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.106+0000 7fe36c790640 10 monclient: monmap: 2024-03-25T18:14:50.107 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:50.107 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:50.107 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:14:50.107642+0000 2024-03-25T18:14:50.107 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:14:50.107642+0000 2024-03-25T18:14:50.108 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:50.108 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:50.108 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:50.108 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:50.108 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:50.108 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:50.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.107+0000 7fe36c790640 1 Processor -- start 2024-03-25T18:14:50.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.107+0000 7fe36c790640 1 -- start start 2024-03-25T18:14:50.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.107+0000 7fe36c790640 10 monclient: init 2024-03-25T18:14:50.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:50.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:50.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 10 monclient(hunting): picked mon.noname-b con 0x7fe3640b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:50.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 10 monclient(hunting): picked mon.noname-a con 0x7fe3640b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:50.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 10 monclient(hunting): picked mon.noname-c con 0x7fe36414be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:50.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3640af940 con 0x7fe3640b1730 2024-03-25T18:14:50.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3640b0a70 con 0x7fe36414be90 2024-03-25T18:14:50.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3640b0d50 con 0x7fe3640b4d60 2024-03-25T18:14:50.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:50.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.108+0000 7fe36c790640 10 monclient(hunting): authenticate will time out at 7589.381925s 2024-03-25T18:14:50.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.115+0000 7fe369d2b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe3640b1730 0x7fe3640b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:49738/0 (socket says 172.21.15.162:49738) 2024-03-25T18:14:50.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.115+0000 7fe369d2b640 1 -- 172.21.15.162:0/709425612 learned_addr learned my addr 172.21.15.162:0/709425612 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:50.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.134+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643588875 0 0) 0x7fe3640af940 con 0x7fe3640b1730 2024-03-25T18:14:50.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.134+0000 7fe36952a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.134+0000 7fe36952a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.134+0000 7fe36952a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.134+0000 7fe36952a640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:14:50.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.134+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe350003520 con 0x7fe3640b1730 2024-03-25T18:14:50.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.136+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185717071 0 0) 0x7fe3640b0a70 con 0x7fe36414be90 2024-03-25T18:14:50.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.136+0000 7fe36952a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.136+0000 7fe36952a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.136+0000 7fe36952a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.136+0000 7fe36952a640 10 monclient(hunting): my global_id is 4240 2024-03-25T18:14:50.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.136+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3640af940 con 0x7fe36414be90 2024-03-25T18:14:50.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 558882412 0 0) 0x7fe3640af940 con 0x7fe36414be90 2024-03-25T18:14:50.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 >> v1:172.21.15.162:6790/0 conn(0x7fe3640b4d60 legacy=0x7fe36413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 >> v1:172.21.15.162:6789/0 conn(0x7fe3640b1730 legacy=0x7fe3640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:50.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 1 monclient: found mon.noname-c 2024-03-25T18:14:50.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3640b1030 con 0x7fe36414be90 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 10 monclient: _finish_auth 0 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe35c0019f0 con 0x7fe36414be90 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36c790640 5 monclient: authenticate success, global_id 4240 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36c790640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36952a640 10 monclient: dump: 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.156 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.157 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.157 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:50.157 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:50.157 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:50.157 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:50.157 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:50.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.154+0000 7fe36c790640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:50.179 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.178+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe35c001b50 con 0x7fe36414be90 2024-03-25T18:14:50.179 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.178+0000 7fe36952a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.178+0000 7fe36952a640 1 -- 172.21.15.162:0/709425612 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe35c001f20 con 0x7fe36414be90 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.178+0000 7fe36952a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.178+0000 7fe36952a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.178+0000 7fe36952a640 10 monclient: dump: 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:50.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.178+0000 7fe36c790640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:50.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.178+0000 7fe36c790640 4 set_mon_vals no callback set 2024-03-25T18:14:50.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.179+0000 7fe36c790640 10 monclient: shutdown 2024-03-25T18:14:50.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.179+0000 7fe36c790640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:50.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.179+0000 7fe36c790640 1 -- 172.21.15.162:0/709425612 >> v1:172.21.15.191:6789/0 conn(0x7fe36414be90 legacy=0x7fe36414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.179+0000 7fe36c790640 1 -- 172.21.15.162:0/709425612 shutdown_connections 2024-03-25T18:14:50.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.179+0000 7fe36c790640 1 -- 172.21.15.162:0/709425612 >> 172.21.15.162:0/709425612 conn(0x7fe3640617a0 msgr2=0x7fe364061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:50.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.179+0000 7fe36c790640 1 -- 172.21.15.162:0/709425612 shutdown_connections 2024-03-25T18:14:50.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.179+0000 7fe36c790640 1 -- 172.21.15.162:0/709425612 wait complete. 2024-03-25T18:14:50.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.180+0000 7fe36c790640 10 monclient: build_initial_monmap 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.180+0000 7fe36c790640 10 monclient: monmap: 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.181+0000 7fe36c790640 1 librados: starting msgr at 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.181+0000 7fe36c790640 1 librados: starting objecter 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.181+0000 7fe36c790640 1 Processor -- start 2024-03-25T18:14:50.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.181+0000 7fe36c790640 1 -- start start 2024-03-25T18:14:50.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.181+0000 7fe36c790640 1 librados: setting wanted keys 2024-03-25T18:14:50.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.181+0000 7fe36c790640 1 librados: calling monclient init 2024-03-25T18:14:50.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.181+0000 7fe36c790640 10 monclient: init 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 10 monclient(hunting): picked mon.noname-a con 0x7fe3640b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 10 monclient(hunting): picked mon.noname-c con 0x7fe3640b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 10 monclient(hunting): picked mon.noname-b con 0x7fe36414be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36413b780 con 0x7fe3640b1730 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36406ec70 con 0x7fe36414be90 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36413b380 con 0x7fe3640b4d60 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:50.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.182+0000 7fe36c790640 10 monclient(hunting): authenticate will time out at 7589.456018s 2024-03-25T18:14:50.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.186+0000 7fe36ad2d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe36414be90 0x7fe364137580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:50358/0 (socket says 172.21.15.162:50358) 2024-03-25T18:14:50.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.186+0000 7fe36ad2d640 1 -- 172.21.15.162:0/3425315227 learned_addr learned my addr 172.21.15.162:0/3425315227 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:50.188 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.187+0000 7fe35affd640 10 client.?.objecter ms_handle_connect 0x7fe36414be90 2024-03-25T18:14:50.188 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.187+0000 7fe35affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:50.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.191+0000 7fe35affd640 10 client.?.objecter ms_handle_connect 0x7fe3640b4d60 2024-03-25T18:14:50.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.191+0000 7fe35affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:50.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.198+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39792737 0 0) 0x7fe36406ec70 con 0x7fe36414be90 2024-03-25T18:14:50.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.198+0000 7fe35affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.198+0000 7fe35affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.198+0000 7fe35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.198+0000 7fe35affd640 10 monclient(hunting): my global_id is 4243 2024-03-25T18:14:50.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.198+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe348004eb0 con 0x7fe36414be90 2024-03-25T18:14:50.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.202+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628152416 0 0) 0x7fe36413b380 con 0x7fe3640b4d60 2024-03-25T18:14:50.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.202+0000 7fe35affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.202+0000 7fe35affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.202+0000 7fe35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.202+0000 7fe35affd640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:14:50.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.202+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36406ec70 con 0x7fe3640b4d60 2024-03-25T18:14:50.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.207+0000 7fe35affd640 10 client.?.objecter ms_handle_connect 0x7fe3640b1730 2024-03-25T18:14:50.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.207+0000 7fe35affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.214+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3276120500 0 0) 0x7fe348004eb0 con 0x7fe36414be90 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.214+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe36413b380 con 0x7fe36414be90 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.214+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe35c001f60 con 0x7fe36414be90 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.214+0000 7fe35affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.214+0000 7fe35affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.214+0000 7fe35affd640 10 monclient(hunting): dump: 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.215 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:50.216 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:50.216 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:50.216 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:50.216 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:50.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.214+0000 7fe35affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:50.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.229+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1496954887 0 0) 0x7fe36406ec70 con 0x7fe3640b4d60 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.229+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe348004eb0 con 0x7fe3640b4d60 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.229+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe354001950 con 0x7fe3640b4d60 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.229+0000 7fe35affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.229+0000 7fe35affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.229+0000 7fe35affd640 10 monclient(hunting): dump: 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:50.231 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:50.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.231+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121748713 0 0) 0x7fe36413b780 con 0x7fe3640b1730 2024-03-25T18:14:50.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.231+0000 7fe35affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.231+0000 7fe35affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.231+0000 7fe35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.231+0000 7fe35affd640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:14:50.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.231+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36406ec70 con 0x7fe3640b1730 2024-03-25T18:14:50.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2885243051 0 0) 0x7fe36413b380 con 0x7fe36414be90 2024-03-25T18:14:50.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 >> v1:172.21.15.162:6790/0 conn(0x7fe3640b4d60 legacy=0x7fe3641dee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 >> v1:172.21.15.162:6789/0 conn(0x7fe3640b1730 legacy=0x7fe36413ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:50.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 1 monclient: found mon.b 2024-03-25T18:14:50.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:50.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36413bc10 con 0x7fe36414be90 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe35affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 5 monclient: authenticate success, global_id 4243 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 10 monclient: _renew_subs 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3641df580 con 0x7fe36414be90 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 10 monclient: _renew_subs 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe364137e70 con 0x7fe36414be90 2024-03-25T18:14:50.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.240+0000 7fe36c790640 1 librados: init done 2024-03-25T18:14:50.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.241+0000 7fe36c790640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:50.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.241+0000 7fe36c790640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:50.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.241+0000 7fe36c790640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:50.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.241+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36413b510 con 0x7fe36414be90 2024-03-25T18:14:50.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.249+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe35c003c20 con 0x7fe36414be90 2024-03-25T18:14:50.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.249+0000 7fe35affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:50.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.249+0000 7fe368d29640 4 set_mon_vals no callback set 2024-03-25T18:14:50.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.290+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe35c003fb0 con 0x7fe36414be90 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.290+0000 7fe35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.290+0000 7fe35affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.290+0000 7fe35affd640 10 monclient: dump: 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:50.291 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:50.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.523+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fe35c004190 con 0x7fe36414be90 2024-03-25T18:14:50.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.524+0000 7fe36a52c640 10 monclient: get_auth_request con 0x7fe34806df50 auth_method 0 2024-03-25T18:14:50.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.524+0000 7fe35affd640 10 client.4243.objecter ms_handle_connect 0x7fe34806df50 2024-03-25T18:14:50.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.527+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fe35c054b10 con 0x7fe36414be90 2024-03-25T18:14:50.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.527+0000 7fe35affd640 10 client.4243.objecter ms_dispatch 0x7fe364000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:50.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.527+0000 7fe35affd640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:50.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.527+0000 7fe35affd640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:50.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.527+0000 7fe35affd640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-25T18:14:50.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.531+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe35c050620 con 0x7fe36414be90 2024-03-25T18:14:50.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.531+0000 7fe35affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:50.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.531+0000 7fe35affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:50.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.787+0000 7fe36c790640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:50.789 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.787+0000 7fe36c790640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:50.789 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.787+0000 7fe36c790640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:50.789 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.787+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe36413b930 con 0x7fe36414be90 2024-03-25T18:14:50.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.814+0000 7fe35affd640 1 -- 172.21.15.162:0/3425315227 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fe36413b930 con 0x7fe36414be90 2024-03-25T18:14:50.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.814+0000 7fe35affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:50.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.814+0000 7fe35affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:50.815 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:14:50.815 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:50.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 10 librados: watch_flush enter 2024-03-25T18:14:50.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 10 librados: watch_flush exit 2024-03-25T18:14:50.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-25T18:14:50.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 10 client.4243.objecter successfully canceled tick 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 >> v1:172.21.15.191:6800/1552703795 conn(0x7fe34806df50 legacy=0x7fe348070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 10 monclient: shutdown 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 >> v1:172.21.15.191:6789/0 conn(0x7fe36414be90 legacy=0x7fe364137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 shutdown_connections 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 >> 172.21.15.162:0/3425315227 conn(0x7fe3640617a0 msgr2=0x7fe3641383c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 shutdown_connections 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 1 -- 172.21.15.162:0/3425315227 wait complete. 2024-03-25T18:14:50.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:50.816+0000 7fe36c790640 1 librados: shutdown 2024-03-25T18:14:50.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:57.637 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:57.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.753+0000 7f82cfb0e640 10 monclient: get_monmap_and_config 2024-03-25T18:14:57.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.753+0000 7f82cfb0e640 10 monclient: build_initial_monmap 2024-03-25T18:14:57.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.753+0000 7f82cfb0e640 10 monclient: monmap: 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:14:57.754850+0000 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:14:57.754850+0000 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:14:57.755 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:57.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.754+0000 7f82cfb0e640 1 Processor -- start 2024-03-25T18:14:57.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.754+0000 7f82cfb0e640 1 -- start start 2024-03-25T18:14:57.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.754+0000 7f82cfb0e640 10 monclient: init 2024-03-25T18:14:57.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f82c80b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f82c80b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f82c814be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80af940 con 0x7f82c814be90 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80b0a70 con 0x7f82c80b4d60 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c80b0d50 con 0x7f82c80b1730 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:57.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.755+0000 7f82cfb0e640 10 monclient(hunting): authenticate will time out at 7597.028923s 2024-03-25T18:14:57.767 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.766+0000 7f82cd8aa640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f82c80b4d60 0x7f82c813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:36502/0 (socket says 172.21.15.162:36502) 2024-03-25T18:14:57.767 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.766+0000 7f82cd8aa640 1 -- 172.21.15.162:0/4182740999 learned_addr learned my addr 172.21.15.162:0/4182740999 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:57.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.776+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194032358 0 0) 0x7f82c80b0a70 con 0x7f82c80b4d60 2024-03-25T18:14:57.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.776+0000 7f82cc8a8640 10 monclient(hunting): _negotiate 2024-03-25T18:14:57.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.776+0000 7f82cc8a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:57.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.776+0000 7f82cc8a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:57.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.776+0000 7f82cc8a8640 10 monclient(hunting): my global_id is 4246 2024-03-25T18:14:57.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.776+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b4003520 con 0x7f82c80b4d60 2024-03-25T18:14:57.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.787+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345668972 0 0) 0x7f82c80af940 con 0x7f82c814be90 2024-03-25T18:14:57.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.787+0000 7f82cc8a8640 10 monclient(hunting): _negotiate 2024-03-25T18:14:57.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.787+0000 7f82cc8a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:57.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.787+0000 7f82cc8a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:57.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.787+0000 7f82cc8a8640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:14:57.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.787+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c80b0a70 con 0x7f82c814be90 2024-03-25T18:14:57.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.793+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856094139 0 0) 0x7f82c80b0d50 con 0x7f82c80b1730 2024-03-25T18:14:57.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.793+0000 7f82cc8a8640 10 monclient(hunting): _negotiate 2024-03-25T18:14:57.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.793+0000 7f82cc8a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:57.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.793+0000 7f82cc8a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:57.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.793+0000 7f82cc8a8640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:14:57.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.793+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82c80af940 con 0x7f82c80b1730 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.796+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541539629 0 0) 0x7f82b4003520 con 0x7f82c80b4d60 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.796+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 >> v1:172.21.15.162:6790/0 conn(0x7f82c80b1730 legacy=0x7f82c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.796+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 >> v1:172.21.15.162:6789/0 conn(0x7f82c814be90 legacy=0x7f82c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.796+0000 7f82cc8a8640 10 monclient: _finish_hunting 0 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 1 monclient: found mon.noname-c 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80b1030 con 0x7f82c80b4d60 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 10 monclient: _finish_auth 0 2024-03-25T18:14:57.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 10 monclient: _check_auth_tickets 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82c4001690 con 0x7f82c80b4d60 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cc8a8640 10 monclient: dump: 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:57.799 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:57.800 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:57.800 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:57.800 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:57.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cfb0e640 5 monclient: authenticate success, global_id 4246 2024-03-25T18:14:57.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.797+0000 7f82cfb0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.804+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c40017f0 con 0x7f82c80b4d60 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.804+0000 7f82cc8a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.804+0000 7f82cc8a8640 1 -- 172.21.15.162:0/4182740999 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82c4001c00 con 0x7f82c80b4d60 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.804+0000 7f82cfb0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.804+0000 7f82cfb0e640 4 set_mon_vals no callback set 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.804+0000 7f82cc8a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.804+0000 7f82cc8a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.804+0000 7f82cc8a8640 10 monclient: dump: 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:57.805 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:57.806 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:57.806 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:57.806 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:57.806 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:57.806 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:57.806 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:57.806 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 10 monclient: shutdown 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 1 -- 172.21.15.162:0/4182740999 >> v1:172.21.15.191:6789/0 conn(0x7f82c80b4d60 legacy=0x7f82c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 1 -- 172.21.15.162:0/4182740999 shutdown_connections 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 1 -- 172.21.15.162:0/4182740999 >> 172.21.15.162:0/4182740999 conn(0x7f82c80617a0 msgr2=0x7f82c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 1 -- 172.21.15.162:0/4182740999 shutdown_connections 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 1 -- 172.21.15.162:0/4182740999 wait complete. 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 10 monclient: build_initial_monmap 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.805+0000 7f82cfb0e640 10 monclient: monmap: 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:14:57.808 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 1 librados: starting msgr at 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 1 librados: starting objecter 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 1 Processor -- start 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 1 -- start start 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 1 librados: setting wanted keys 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 1 librados: calling monclient init 2024-03-25T18:14:57.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 10 monclient: init 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 10 monclient(hunting): picked mon.noname-b con 0x7f82c80b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 10 monclient(hunting): picked mon.noname-a con 0x7f82c80b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 10 monclient(hunting): picked mon.noname-c con 0x7f82c814be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c81d7730 con 0x7f82c80b4d60 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.806+0000 7f82cfb0e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c813bdf0 con 0x7f82c80b1730 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.807+0000 7f82cfb0e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c8142100 con 0x7f82c814be90 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.807+0000 7f82cfb0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:57.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.807+0000 7f82cfb0e640 10 monclient(hunting): authenticate will time out at 7597.080670s 2024-03-25T18:14:57.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.816+0000 7f82cd8aa640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f82c80b1730 0x7f82c81405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:36542/0 (socket says 172.21.15.162:36542) 2024-03-25T18:14:57.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.816+0000 7f82cd8aa640 1 -- 172.21.15.162:0/2422901492 learned_addr learned my addr 172.21.15.162:0/2422901492 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:14:57.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.817+0000 7f82be7fc640 10 client.?.objecter ms_handle_connect 0x7f82c80b1730 2024-03-25T18:14:57.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.817+0000 7f82be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:57.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.821+0000 7f82be7fc640 10 client.?.objecter ms_handle_connect 0x7f82c80b4d60 2024-03-25T18:14:57.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.821+0000 7f82be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:57.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.822+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059529590 0 0) 0x7f82c813bdf0 con 0x7f82c80b1730 2024-03-25T18:14:57.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.822+0000 7f82be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:57.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.822+0000 7f82be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:57.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.822+0000 7f82be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:57.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.822+0000 7f82be7fc640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:14:57.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.822+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82ac004eb0 con 0x7f82c80b1730 2024-03-25T18:14:57.824 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.823+0000 7f82be7fc640 10 client.?.objecter ms_handle_connect 0x7f82c814be90 2024-03-25T18:14:57.824 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.823+0000 7f82be7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:57.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.830+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3371958025 0 0) 0x7f82ac004eb0 con 0x7f82c80b1730 2024-03-25T18:14:57.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.830+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82c813bdf0 con 0x7f82c80b1730 2024-03-25T18:14:57.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.830+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82c4001bc0 con 0x7f82c80b1730 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.830+0000 7f82be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.830+0000 7f82be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.830+0000 7f82be7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:57.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.830+0000 7f82be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:57.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 123864052 0 0) 0x7f82c813bdf0 con 0x7f82c80b1730 2024-03-25T18:14:57.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 >> v1:172.21.15.162:6790/0 conn(0x7f82c814be90 legacy=0x7f82c81d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 >> v1:172.21.15.162:6789/0 conn(0x7f82c80b4d60 legacy=0x7f82c813b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 1 monclient: found mon.b 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c8142290 con 0x7f82c80b1730 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 5 monclient: authenticate success, global_id 4249 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 10 monclient: _renew_subs 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c81d8320 con 0x7f82c80b1730 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:57.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 10 monclient: _renew_subs 2024-03-25T18:14:57.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:57.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c81d85d0 con 0x7f82c80b1730 2024-03-25T18:14:57.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.834+0000 7f82cfb0e640 1 librados: init done 2024-03-25T18:14:57.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.835+0000 7f82cfb0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:57.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.835+0000 7f82cfb0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:57.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.835+0000 7f82cfb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:57.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:57.835+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c80627b0 con 0x7f82c80b1730 2024-03-25T18:14:58.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.110+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82c40011f0 con 0x7f82c80b1730 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.110+0000 7f82be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.110+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82c4004c70 con 0x7f82c80b1730 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.110+0000 7f82be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.110+0000 7f82be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.110+0000 7f82be7fc640 10 monclient: dump: 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:14:58.112 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:14:58.113 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:14:58.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.110+0000 7f82bffff640 4 set_mon_vals no callback set 2024-03-25T18:14:58.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.111+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f82c4004e90 con 0x7f82c80b1730 2024-03-25T18:14:58.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.112+0000 7f82cd0a9640 10 monclient: get_auth_request con 0x7f82ac06dc00 auth_method 0 2024-03-25T18:14:58.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.112+0000 7f82be7fc640 10 client.4249.objecter ms_handle_connect 0x7f82ac06dc00 2024-03-25T18:14:58.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.116+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f82c4054970 con 0x7f82c80b1730 2024-03-25T18:14:58.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.116+0000 7f82be7fc640 10 client.4249.objecter ms_dispatch 0x7f82c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:58.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.116+0000 7f82be7fc640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:58.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.116+0000 7f82be7fc640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:58.118 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.116+0000 7f82be7fc640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-25T18:14:58.118 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.117+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82c40502b0 con 0x7f82c80b1730 2024-03-25T18:14:58.118 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.117+0000 7f82be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:58.118 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.117+0000 7f82be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:58.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.369+0000 7f82cfb0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:58.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.369+0000 7f82cfb0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:58.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.369+0000 7f82cfb0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:14:58.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.369+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82c8140170 con 0x7f82c80b1730 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.388+0000 7f82be7fc640 1 -- 172.21.15.162:0/2422901492 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f82c40523b0 con 0x7f82c80b1730 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.388+0000 7f82be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.388+0000 7f82be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 10 librados: watch_flush enter 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 10 librados: watch_flush exit 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 10 client.4249.objecter successfully canceled tick 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 >> v1:172.21.15.191:6800/1552703795 conn(0x7f82ac06dc00 legacy=0x7f82ac0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 10 monclient: shutdown 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 >> v1:172.21.15.191:6789/0 conn(0x7f82c80b1730 legacy=0x7f82c81405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 shutdown_connections 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 >> 172.21.15.162:0/2422901492 conn(0x7f82c80617a0 msgr2=0x7f82c80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 shutdown_connections 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 1 -- 172.21.15.162:0/2422901492 wait complete. 2024-03-25T18:14:58.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:14:58.391+0000 7f82cfb0e640 1 librados: shutdown 2024-03-25T18:14:58.404 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:02.085 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:14.313 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.144 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:14.372 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.178 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:14.394 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.206 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:14.434 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.238 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:14.454 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.268 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:14.495 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.388 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:14.615 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.450 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:14.666 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:02.510 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:14.737 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:05.211 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:05.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.327+0000 7f15f4cc3640 10 monclient: get_monmap_and_config 2024-03-25T18:15:05.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.327+0000 7f15f4cc3640 10 monclient: build_initial_monmap 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.327+0000 7f15f4cc3640 10 monclient: monmap: 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:15:05.328346+0000 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:15:05.328346+0000 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.328+0000 7f15f4cc3640 1 Processor -- start 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.328+0000 7f15f4cc3640 1 -- start start 2024-03-25T18:15:05.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.328+0000 7f15f4cc3640 10 monclient: init 2024-03-25T18:15:05.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:05.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:05.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 10 monclient(hunting): picked mon.noname-a con 0x7f15f00a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:05.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 10 monclient(hunting): picked mon.noname-c con 0x7f15f013f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:05.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 10 monclient(hunting): picked mon.noname-b con 0x7f15f01430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:05.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f0142520 con 0x7f15f00a98f0 2024-03-25T18:15:05.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00b28a0 con 0x7f15f013f710 2024-03-25T18:15:05.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00b2a30 con 0x7f15f01430c0 2024-03-25T18:15:05.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:05.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.329+0000 7f15f4cc3640 10 monclient(hunting): authenticate will time out at 7604.602901s 2024-03-25T18:15:05.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.342+0000 7f15edd9b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f15f013f710 0x7f15f01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:58730/0 (socket says 172.21.15.162:58730) 2024-03-25T18:15:05.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.342+0000 7f15edd9b640 1 -- 172.21.15.162:0/2704418428 learned_addr learned my addr 172.21.15.162:0/2704418428 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:05.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.352+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287398076 0 0) 0x7f15f00b28a0 con 0x7f15f013f710 2024-03-25T18:15:05.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.352+0000 7f15ed59a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:05.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.352+0000 7f15ed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:05.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.352+0000 7f15ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:05.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.352+0000 7f15ed59a640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:15:05.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.352+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d8003520 con 0x7f15f013f710 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4217612276 0 0) 0x7f15d8003520 con 0x7f15f013f710 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 >> v1:172.21.15.162:6790/0 conn(0x7f15f01430c0 legacy=0x7f15f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 >> v1:172.21.15.162:6789/0 conn(0x7f15f00a98f0 legacy=0x7f15f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 10 monclient: _finish_hunting 0 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 1 monclient: found mon.noname-c 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f00b44c0 con 0x7f15f013f710 2024-03-25T18:15:05.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 10 monclient: _finish_auth 0 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 10 monclient: _check_auth_tickets 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f15e4001690 con 0x7f15f013f710 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15f4cc3640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15f4cc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15ed59a640 10 monclient: dump: 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:05.364 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:05.365 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:05.365 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:05.365 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.362+0000 7f15f4cc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:05.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.367+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15e40017f0 con 0x7f15f013f710 2024-03-25T18:15:05.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.367+0000 7f15ed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:05.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.367+0000 7f15ed59a640 1 -- 172.21.15.162:0/2704418428 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f15e4001c00 con 0x7f15f013f710 2024-03-25T18:15:05.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.367+0000 7f15f4cc3640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:05.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.367+0000 7f15f4cc3640 4 set_mon_vals no callback set 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.367+0000 7f15ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.367+0000 7f15ed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.367+0000 7f15ed59a640 10 monclient: dump: 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:05.369 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.368+0000 7f15f4cc3640 10 monclient: shutdown 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.368+0000 7f15f4cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.368+0000 7f15f4cc3640 1 -- 172.21.15.162:0/2704418428 >> v1:172.21.15.191:6789/0 conn(0x7f15f013f710 legacy=0x7f15f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.368+0000 7f15f4cc3640 1 -- 172.21.15.162:0/2704418428 shutdown_connections 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.368+0000 7f15f4cc3640 1 -- 172.21.15.162:0/2704418428 >> 172.21.15.162:0/2704418428 conn(0x7f15f00b7ea0 msgr2=0x7f15f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.368+0000 7f15f4cc3640 1 -- 172.21.15.162:0/2704418428 shutdown_connections 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.368+0000 7f15f4cc3640 1 -- 172.21.15.162:0/2704418428 wait complete. 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.369+0000 7f15f4cc3640 10 monclient: build_initial_monmap 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.369+0000 7f15f4cc3640 10 monclient: monmap: 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:15:05.371 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 1 librados: starting msgr at 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 1 librados: starting objecter 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 1 Processor -- start 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 1 -- start start 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 1 librados: setting wanted keys 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 1 librados: calling monclient init 2024-03-25T18:15:05.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 10 monclient: init 2024-03-25T18:15:05.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:05.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:05.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 10 monclient(hunting): picked mon.noname-b con 0x7f15f00a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:05.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.370+0000 7f15f4cc3640 10 monclient(hunting): picked mon.noname-a con 0x7f15f013f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:05.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.371+0000 7f15f4cc3640 10 monclient(hunting): picked mon.noname-c con 0x7f15f01430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:05.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.371+0000 7f15f4cc3640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00b1870 con 0x7f15f013f710 2024-03-25T18:15:05.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.371+0000 7f15f4cc3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00ac910 con 0x7f15f00a98f0 2024-03-25T18:15:05.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.371+0000 7f15f4cc3640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f00abb00 con 0x7f15f01430c0 2024-03-25T18:15:05.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.371+0000 7f15f4cc3640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:05.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.371+0000 7f15f4cc3640 10 monclient(hunting): authenticate will time out at 7604.644862s 2024-03-25T18:15:05.376 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.375+0000 7f15eed9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f15f01430c0 0x7f15f01e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:52166/0 (socket says 172.21.15.162:52166) 2024-03-25T18:15:05.376 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.375+0000 7f15eed9d640 1 -- 172.21.15.162:0/3933506853 learned_addr learned my addr 172.21.15.162:0/3933506853 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:05.376 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.375+0000 7f15deffd640 10 client.?.objecter ms_handle_connect 0x7f15f01430c0 2024-03-25T18:15:05.376 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.375+0000 7f15deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:05.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.376+0000 7f15deffd640 10 client.?.objecter ms_handle_connect 0x7f15f00a98f0 2024-03-25T18:15:05.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.376+0000 7f15deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:05.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.377+0000 7f15deffd640 10 client.?.objecter ms_handle_connect 0x7f15f013f710 2024-03-25T18:15:05.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.377+0000 7f15deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:05.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.381+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346070797 0 0) 0x7f15f00abb00 con 0x7f15f01430c0 2024-03-25T18:15:05.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.381+0000 7f15deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:05.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.381+0000 7f15deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:05.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.381+0000 7f15deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:05.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.381+0000 7f15deffd640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:15:05.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.381+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15d0004eb0 con 0x7f15f01430c0 2024-03-25T18:15:05.383 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.383+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196455099 0 0) 0x7f15f00ac910 con 0x7f15f00a98f0 2024-03-25T18:15:05.383 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.383+0000 7f15deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:05.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.383+0000 7f15deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:05.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.383+0000 7f15deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:05.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.383+0000 7f15deffd640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:15:05.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.383+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f00abb00 con 0x7f15f00a98f0 2024-03-25T18:15:05.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.389+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450215056 0 0) 0x7f15f00b44c0 con 0x7f15f013f710 2024-03-25T18:15:05.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.389+0000 7f15deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:05.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.389+0000 7f15deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:05.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.389+0000 7f15deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:05.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.389+0000 7f15deffd640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:15:05.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.389+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15f00ac910 con 0x7f15f013f710 2024-03-25T18:15:05.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.391+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 397101213 0 0) 0x7f15d0004eb0 con 0x7f15f01430c0 2024-03-25T18:15:05.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.391+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15f00b44c0 con 0x7f15f01430c0 2024-03-25T18:15:05.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.391+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f15e8001c30 con 0x7f15f01430c0 2024-03-25T18:15:05.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.391+0000 7f15deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:05.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.391+0000 7f15deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:05.392 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.391+0000 7f15deffd640 10 monclient(hunting): dump: 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:05.393 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.391+0000 7f15deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.397+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3858246012 0 0) 0x7f15f00ac910 con 0x7f15f013f710 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.397+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15d0004eb0 con 0x7f15f013f710 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.397+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f15e4001960 con 0x7f15f013f710 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.397+0000 7f15deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.397+0000 7f15deffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.397+0000 7f15deffd640 10 monclient(hunting): dump: 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:05.398 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:05.399 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:05.399 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:05.399 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:05.399 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2253951807 0 0) 0x7f15f00b44c0 con 0x7f15f01430c0 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 >> v1:172.21.15.191:6789/0 conn(0x7f15f00a98f0 legacy=0x7f15f0138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 >> v1:172.21.15.162:6789/0 conn(0x7f15f013f710 legacy=0x7f15f00ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 1 monclient: found mon.c 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f00abde0 con 0x7f15f01430c0 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:05.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15f4cc3640 5 monclient: authenticate success, global_id 4265 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15f4cc3640 10 monclient: _renew_subs 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15f4cc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.399+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f0138f40 con 0x7f15f01430c0 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.400+0000 7f15f4cc3640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.400+0000 7f15f4cc3640 10 monclient: _renew_subs 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.400+0000 7f15f4cc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.400+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f01363f0 con 0x7f15f01430c0 2024-03-25T18:15:05.402 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.400+0000 7f15f4cc3640 1 librados: init done 2024-03-25T18:15:05.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.401+0000 7f15f4cc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:05.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.401+0000 7f15f4cc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:05.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.401+0000 7f15f4cc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:05.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.401+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f00627b0 con 0x7f15f01430c0 2024-03-25T18:15:05.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.681+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15e8003990 con 0x7f15f01430c0 2024-03-25T18:15:05.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.681+0000 7f15deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:05.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.681+0000 7f15ecd99640 4 set_mon_vals no callback set 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.690+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f15e8003da0 con 0x7f15f01430c0 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.690+0000 7f15deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.690+0000 7f15deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.690+0000 7f15deffd640 10 monclient: dump: 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:05.691 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:05.692 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:05.692 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:05.692 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:05.692 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:05.692 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.691+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f15e8003f80 con 0x7f15f01430c0 2024-03-25T18:15:05.692 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.691+0000 7f15ee59c640 10 monclient: get_auth_request con 0x7f15d006df20 auth_method 0 2024-03-25T18:15:05.692 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.691+0000 7f15deffd640 10 client.4265.objecter ms_handle_connect 0x7f15d006df20 2024-03-25T18:15:05.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.696+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f15e8054af0 con 0x7f15f01430c0 2024-03-25T18:15:05.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.696+0000 7f15deffd640 10 client.4265.objecter ms_dispatch 0x7f15f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:05.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.696+0000 7f15deffd640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:05.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.696+0000 7f15deffd640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:05.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.696+0000 7f15deffd640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-25T18:15:05.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.696+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15e8050600 con 0x7f15f01430c0 2024-03-25T18:15:05.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.696+0000 7f15deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:05.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.696+0000 7f15deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:05.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.935+0000 7f15f4cc3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:05.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.935+0000 7f15f4cc3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:05.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.935+0000 7f15f4cc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:05.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.935+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15f01434a0 con 0x7f15f01430c0 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.957+0000 7f15deffd640 1 -- 172.21.15.162:0/3933506853 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f15e8052530 con 0x7f15f01430c0 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.958+0000 7f15deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.958+0000 7f15deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 10 librados: watch_flush enter 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 10 librados: watch_flush exit 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 10 client.4265.objecter successfully canceled tick 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 >> v1:172.21.15.191:6800/1552703795 conn(0x7f15d006df20 legacy=0x7f15d00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 10 monclient: shutdown 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 >> v1:172.21.15.162:6790/0 conn(0x7f15f01430c0 legacy=0x7f15f01e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 shutdown_connections 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 >> 172.21.15.162:0/3933506853 conn(0x7f15f00b7ea0 msgr2=0x7f15f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 shutdown_connections 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 1 -- 172.21.15.162:0/3933506853 wait complete. 2024-03-25T18:15:05.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:05.960+0000 7f15f4cc3640 1 librados: shutdown 2024-03-25T18:15:05.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:12.777 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.893+0000 7f6e8c72b640 10 monclient: get_monmap_and_config 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.893+0000 7f6e8c72b640 10 monclient: build_initial_monmap 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.894+0000 7f6e8c72b640 10 monclient: monmap: 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:15:12.895048+0000 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:15:12.895048+0000 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:15:12.895 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:12.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.894+0000 7f6e8c72b640 1 Processor -- start 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.894+0000 7f6e8c72b640 1 -- start start 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.894+0000 7f6e8c72b640 10 monclient: init 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 10 monclient(hunting): picked mon.noname-b con 0x7f6e84061830 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8413c9f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 10 monclient(hunting): picked mon.noname-a con 0x7f6e84072af0 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e84071fc0 con 0x7f6e84072af0 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840707a0 con 0x7f6e8413c9f0 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840ae9e0 con 0x7f6e84061830 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:12.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.895+0000 7f6e8c72b640 10 monclient(hunting): authenticate will time out at 7612.169414s 2024-03-25T18:15:12.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.900+0000 7f6e8acc8640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6e84072af0 0x7f6e8414a890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:50352/0 (socket says 172.21.15.162:50352) 2024-03-25T18:15:12.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.900+0000 7f6e8acc8640 1 -- 172.21.15.162:0/3087386098 learned_addr learned my addr 172.21.15.162:0/3087386098 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:12.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.912+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526112157 0 0) 0x7f6e84071fc0 con 0x7f6e84072af0 2024-03-25T18:15:12.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.912+0000 7f6e894c5640 10 monclient(hunting): _negotiate 2024-03-25T18:15:12.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.912+0000 7f6e894c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:12.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.912+0000 7f6e894c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:12.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.912+0000 7f6e894c5640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:15:12.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.912+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e70003520 con 0x7f6e84072af0 2024-03-25T18:15:12.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.913+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187624261 0 0) 0x7f6e840707a0 con 0x7f6e8413c9f0 2024-03-25T18:15:12.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.913+0000 7f6e894c5640 10 monclient(hunting): _negotiate 2024-03-25T18:15:12.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.913+0000 7f6e894c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:12.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.913+0000 7f6e894c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:12.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.913+0000 7f6e894c5640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:15:12.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.913+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84071fc0 con 0x7f6e8413c9f0 2024-03-25T18:15:12.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135601531 0 0) 0x7f6e84071fc0 con 0x7f6e8413c9f0 2024-03-25T18:15:12.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 >> v1:172.21.15.162:6790/0 conn(0x7f6e84061830 legacy=0x7f6e8413c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:15:12.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 >> v1:172.21.15.162:6789/0 conn(0x7f6e84072af0 legacy=0x7f6e8414a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:12.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 10 monclient: _finish_hunting 0 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 1 monclient: found mon.noname-c 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84070cc0 con 0x7f6e8413c9f0 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 10 monclient: _finish_auth 0 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 10 monclient: _check_auth_tickets 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6e74001670 con 0x7f6e8413c9f0 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e894c5640 10 monclient: dump: 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:12.928 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:12.929 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:12.929 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:12.929 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:12.929 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:12.929 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:12.929 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:12.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e8c72b640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:15:12.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.926+0000 7f6e8c72b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:12.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.960+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e740017d0 con 0x7f6e8413c9f0 2024-03-25T18:15:12.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.960+0000 7f6e894c5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:12.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.960+0000 7f6e894c5640 1 -- 172.21.15.162:0/3087386098 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6e74001c20 con 0x7f6e8413c9f0 2024-03-25T18:15:12.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.960+0000 7f6e8c72b640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:12.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.960+0000 7f6e8c72b640 4 set_mon_vals no callback set 2024-03-25T18:15:12.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.960+0000 7f6e894c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:12.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.960+0000 7f6e894c5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.960+0000 7f6e894c5640 10 monclient: dump: 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:12.962 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:12.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.961+0000 7f6e8c72b640 10 monclient: shutdown 2024-03-25T18:15:12.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.961+0000 7f6e8c72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.961+0000 7f6e8c72b640 1 -- 172.21.15.162:0/3087386098 >> v1:172.21.15.191:6789/0 conn(0x7f6e8413c9f0 legacy=0x7f6e840723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.961+0000 7f6e8c72b640 1 -- 172.21.15.162:0/3087386098 shutdown_connections 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.961+0000 7f6e8c72b640 1 -- 172.21.15.162:0/3087386098 >> 172.21.15.162:0/3087386098 conn(0x7f6e840b0ef0 msgr2=0x7f6e840b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.961+0000 7f6e8c72b640 1 -- 172.21.15.162:0/3087386098 shutdown_connections 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.961+0000 7f6e8c72b640 1 -- 172.21.15.162:0/3087386098 wait complete. 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 10 monclient: build_initial_monmap 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 10 monclient: monmap: 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:15:12.964 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 1 librados: starting msgr at 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 1 librados: starting objecter 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 1 Processor -- start 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 1 -- start start 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 1 librados: setting wanted keys 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 1 librados: calling monclient init 2024-03-25T18:15:12.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.962+0000 7f6e8c72b640 10 monclient: init 2024-03-25T18:15:12.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:12.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:12.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 10 monclient(hunting): picked mon.noname-b con 0x7f6e84061830 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:12.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 10 monclient(hunting): picked mon.noname-a con 0x7f6e84072af0 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:12.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8413c9f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:12.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e84070210 con 0x7f6e84072af0 2024-03-25T18:15:12.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840aecf0 con 0x7f6e84061830 2024-03-25T18:15:12.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e84132aa0 con 0x7f6e8413c9f0 2024-03-25T18:15:12.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:12.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.963+0000 7f6e8c72b640 10 monclient(hunting): authenticate will time out at 7612.237320s 2024-03-25T18:15:12.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.967+0000 7f6e89cc6640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6e84072af0 0x7f6e8412c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:50398/0 (socket says 172.21.15.162:50398) 2024-03-25T18:15:12.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.967+0000 7f6e89cc6640 1 -- 172.21.15.162:0/590373013 learned_addr learned my addr 172.21.15.162:0/590373013 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:12.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.968+0000 7f6e7affd640 10 client.?.objecter ms_handle_connect 0x7f6e84072af0 2024-03-25T18:15:12.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.968+0000 7f6e7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:12.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.976+0000 7f6e7affd640 10 client.?.objecter ms_handle_connect 0x7f6e8413c9f0 2024-03-25T18:15:12.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.977+0000 7f6e7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:12.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.977+0000 7f6e7affd640 10 client.?.objecter ms_handle_connect 0x7f6e84061830 2024-03-25T18:15:12.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.977+0000 7f6e7affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:12.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.983+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705839303 0 0) 0x7f6e84070210 con 0x7f6e84072af0 2024-03-25T18:15:12.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.983+0000 7f6e7affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:12.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.983+0000 7f6e7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:12.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.983+0000 7f6e7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:12.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.983+0000 7f6e7affd640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:15:12.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.983+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e68004eb0 con 0x7f6e84072af0 2024-03-25T18:15:12.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.985+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120113864 0 0) 0x7f6e84132aa0 con 0x7f6e8413c9f0 2024-03-25T18:15:12.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.985+0000 7f6e7affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:12.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.985+0000 7f6e7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:12.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.985+0000 7f6e7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:12.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.985+0000 7f6e7affd640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:15:12.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.985+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84070210 con 0x7f6e8413c9f0 2024-03-25T18:15:12.993 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.992+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787548580 0 0) 0x7f6e840aecf0 con 0x7f6e84061830 2024-03-25T18:15:12.993 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.992+0000 7f6e7affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:12.993 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.992+0000 7f6e7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:12.993 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.992+0000 7f6e7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:12.993 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.992+0000 7f6e7affd640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:15:12.993 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:12.992+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84132aa0 con 0x7f6e84061830 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.005+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084913177 0 0) 0x7f6e68004eb0 con 0x7f6e84072af0 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.005+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e840aecf0 con 0x7f6e84072af0 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.005+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6e74001630 con 0x7f6e84072af0 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.005+0000 7f6e7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.005+0000 7f6e7affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.005+0000 7f6e7affd640 10 monclient(hunting): dump: 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:13.006 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:13.007 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:13.007 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.007 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:13.007 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:13.007 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:13.007 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:13.007 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:13.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.005+0000 7f6e7affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.006+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3071112241 0 0) 0x7f6e84132aa0 con 0x7f6e84061830 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.006+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e68004eb0 con 0x7f6e84061830 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.006+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6e800017d0 con 0x7f6e84061830 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.006+0000 7f6e7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.006+0000 7f6e7affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.006+0000 7f6e7affd640 10 monclient(hunting): dump: 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:13.008 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:13.009 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:13.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.017+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1282055850 0 0) 0x7f6e84070210 con 0x7f6e8413c9f0 2024-03-25T18:15:13.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.017+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e84132aa0 con 0x7f6e8413c9f0 2024-03-25T18:15:13.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.017+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6e7c001cd0 con 0x7f6e8413c9f0 2024-03-25T18:15:13.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.017+0000 7f6e7affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.017+0000 7f6e7affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:13.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.017+0000 7f6e7affd640 10 monclient(hunting): dump: 2024-03-25T18:15:13.019 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2588594773 0 0) 0x7f6e68004eb0 con 0x7f6e84061830 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 >> v1:172.21.15.162:6790/0 conn(0x7f6e8413c9f0 legacy=0x7f6e841e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 >> v1:172.21.15.162:6789/0 conn(0x7f6e84072af0 legacy=0x7f6e8412c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.020 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 1 monclient: found mon.b 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e841323a0 con 0x7f6e84061830 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 10 monclient: _finish_auth 0 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 10 monclient: _renew_subs 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e84132680 con 0x7f6e84061830 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 10 monclient: _renew_subs 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:13.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e84133260 con 0x7f6e84061830 2024-03-25T18:15:13.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.018+0000 7f6e8c72b640 1 librados: init done 2024-03-25T18:15:13.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.019+0000 7f6e8c72b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:13.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.019+0000 7f6e8c72b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:13.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.019+0000 7f6e8c72b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:13.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.019+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e840627b0 con 0x7f6e84061830 2024-03-25T18:15:13.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.295+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e80001cc0 con 0x7f6e84061830 2024-03-25T18:15:13.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.295+0000 7f6e7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:13.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.295+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6e80003790 con 0x7f6e84061830 2024-03-25T18:15:13.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.295+0000 7f6e7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.295+0000 7f6e7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.295+0000 7f6e7affd640 10 monclient: dump: 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:13.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.295+0000 7f6e88cc4640 4 set_mon_vals no callback set 2024-03-25T18:15:13.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.296+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f6e80019760 con 0x7f6e84061830 2024-03-25T18:15:13.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.296+0000 7f6e89cc6640 10 monclient: get_auth_request con 0x7f6e6806e080 auth_method 0 2024-03-25T18:15:13.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.297+0000 7f6e7affd640 10 client.4261.objecter ms_handle_connect 0x7f6e6806e080 2024-03-25T18:15:13.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.305+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f6e80053460 con 0x7f6e84061830 2024-03-25T18:15:13.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.305+0000 7f6e7affd640 10 client.4261.objecter ms_dispatch 0x7f6e84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:13.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.305+0000 7f6e7affd640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:13.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.305+0000 7f6e7affd640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:13.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.305+0000 7f6e7affd640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T18:15:13.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.305+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e8004eda0 con 0x7f6e84061830 2024-03-25T18:15:13.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.305+0000 7f6e7affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:13.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.305+0000 7f6e7affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:13.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.555+0000 7f6e8c72b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:13.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.555+0000 7f6e8c72b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:13.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.555+0000 7f6e8c72b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:13.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.555+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e84069f40 con 0x7f6e84061830 2024-03-25T18:15:13.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.654+0000 7f6e7affd640 1 -- 172.21.15.162:0/590373013 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f6e80050ea0 con 0x7f6e84061830 2024-03-25T18:15:13.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.654+0000 7f6e7affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:13.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.654+0000 7f6e7affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:13.655 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:15:13.655 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 10 librados: watch_flush enter 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 10 librados: watch_flush exit 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 10 client.4261.objecter successfully canceled tick 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 >> v1:172.21.15.191:6800/1552703795 conn(0x7f6e6806e080 legacy=0x7f6e68070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 10 monclient: shutdown 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 >> v1:172.21.15.191:6789/0 conn(0x7f6e84061830 legacy=0x7f6e840b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 shutdown_connections 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 >> 172.21.15.162:0/590373013 conn(0x7f6e840b0ef0 msgr2=0x7f6e840b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 shutdown_connections 2024-03-25T18:15:13.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 1 -- 172.21.15.162:0/590373013 wait complete. 2024-03-25T18:15:13.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:13.656+0000 7f6e8c72b640 1 librados: shutdown 2024-03-25T18:15:13.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:20.475 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.590+0000 7fce3236b640 10 monclient: get_monmap_and_config 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.590+0000 7fce3236b640 10 monclient: build_initial_monmap 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.590+0000 7fce3236b640 10 monclient: monmap: 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:15:20.591354+0000 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:15:20.591354+0000 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:20.591 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:20.592 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:15:20.592 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:15:20.592 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:20.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.591+0000 7fce3236b640 1 Processor -- start 2024-03-25T18:15:20.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.591+0000 7fce3236b640 1 -- start start 2024-03-25T18:15:20.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.591+0000 7fce3236b640 10 monclient: init 2024-03-25T18:15:20.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:20.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:20.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 10 monclient(hunting): picked mon.noname-a con 0x7fce2c0a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:20.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 10 monclient(hunting): picked mon.noname-c con 0x7fce2c13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:20.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 10 monclient(hunting): picked mon.noname-b con 0x7fce2c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:20.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c142520 con 0x7fce2c0a98f0 2024-03-25T18:15:20.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c0b28a0 con 0x7fce2c13f710 2024-03-25T18:15:20.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c0b2a30 con 0x7fce2c1430c0 2024-03-25T18:15:20.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:20.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.592+0000 7fce3236b640 10 monclient(hunting): authenticate will time out at 7619.865909s 2024-03-25T18:15:20.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.594+0000 7fce2b7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fce2c13f710 0x7fce2c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:53956/0 (socket says 172.21.15.162:53956) 2024-03-25T18:15:20.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.594+0000 7fce2b7fe640 1 -- 172.21.15.162:0/1610962339 learned_addr learned my addr 172.21.15.162:0/1610962339 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:20.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.604+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745387663 0 0) 0x7fce2c0b28a0 con 0x7fce2c13f710 2024-03-25T18:15:20.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.604+0000 7fce2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:20.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.604+0000 7fce2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:20.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.604+0000 7fce2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:20.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.604+0000 7fce2affd640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:15:20.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.604+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce14003520 con 0x7fce2c13f710 2024-03-25T18:15:20.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1853510811 0 0) 0x7fce14003520 con 0x7fce2c13f710 2024-03-25T18:15:20.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 >> v1:172.21.15.162:6790/0 conn(0x7fce2c1430c0 legacy=0x7fce2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:20.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 >> v1:172.21.15.162:6789/0 conn(0x7fce2c0a98f0 legacy=0x7fce2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:20.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:20.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 1 monclient: found mon.noname-c 2024-03-25T18:15:20.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:20.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:15:20.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c0b44c0 con 0x7fce2c13f710 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 10 monclient: _finish_auth 0 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fce20001690 con 0x7fce2c13f710 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce2affd640 10 monclient: dump: 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:20.629 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:20.630 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:20.630 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:20.630 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce3236b640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:15:20.630 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.627+0000 7fce3236b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.635+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce200017f0 con 0x7fce2c13f710 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.635+0000 7fce2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.635+0000 7fce2affd640 1 -- 172.21.15.162:0/1610962339 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fce20001c00 con 0x7fce2c13f710 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.635+0000 7fce3236b640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.635+0000 7fce3236b640 4 set_mon_vals no callback set 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.635+0000 7fce2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.635+0000 7fce2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.635+0000 7fce2affd640 10 monclient: dump: 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:20.636 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:20.637 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:20.637 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:20.637 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:20.637 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:20.637 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:20.637 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:20.637 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:20.637 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:20.639 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.636+0000 7fce3236b640 10 monclient: shutdown 2024-03-25T18:15:20.639 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.636+0000 7fce3236b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:20.639 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.636+0000 7fce3236b640 1 -- 172.21.15.162:0/1610962339 >> v1:172.21.15.191:6789/0 conn(0x7fce2c13f710 legacy=0x7fce2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.636+0000 7fce3236b640 1 -- 172.21.15.162:0/1610962339 shutdown_connections 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.636+0000 7fce3236b640 1 -- 172.21.15.162:0/1610962339 >> 172.21.15.162:0/1610962339 conn(0x7fce2c0b7ea0 msgr2=0x7fce2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.636+0000 7fce3236b640 1 -- 172.21.15.162:0/1610962339 shutdown_connections 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.636+0000 7fce3236b640 1 -- 172.21.15.162:0/1610962339 wait complete. 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.637+0000 7fce3236b640 10 monclient: build_initial_monmap 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.637+0000 7fce3236b640 10 monclient: monmap: 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:15:20.640 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.637+0000 7fce3236b640 1 librados: starting msgr at 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.637+0000 7fce3236b640 1 librados: starting objecter 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.637+0000 7fce3236b640 1 Processor -- start 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 1 -- start start 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 1 librados: setting wanted keys 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 1 librados: calling monclient init 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 10 monclient: init 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 10 monclient(hunting): picked mon.noname-b con 0x7fce2c0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 10 monclient(hunting): picked mon.noname-a con 0x7fce2c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 10 monclient(hunting): picked mon.noname-c con 0x7fce2c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c137df0 con 0x7fce2c13f710 2024-03-25T18:15:20.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c1337a0 con 0x7fce2c0a98f0 2024-03-25T18:15:20.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c133930 con 0x7fce2c1430c0 2024-03-25T18:15:20.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:20.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.638+0000 7fce3236b640 10 monclient(hunting): authenticate will time out at 7619.912511s 2024-03-25T18:15:20.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.640+0000 7fce30908640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fce2c1430c0 0x7fce2c133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:35826/0 (socket says 172.21.15.162:35826) 2024-03-25T18:15:20.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.640+0000 7fce30908640 1 -- 172.21.15.162:0/1964780225 learned_addr learned my addr 172.21.15.162:0/1964780225 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:20.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.641+0000 7fce28ff9640 10 client.?.objecter ms_handle_connect 0x7fce2c1430c0 2024-03-25T18:15:20.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.641+0000 7fce28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:20.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.642+0000 7fce28ff9640 10 client.?.objecter ms_handle_connect 0x7fce2c0a98f0 2024-03-25T18:15:20.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.642+0000 7fce28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:20.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.718+0000 7fce28ff9640 10 client.?.objecter ms_handle_connect 0x7fce2c13f710 2024-03-25T18:15:20.719 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.718+0000 7fce28ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:20.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.735+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121790005 0 0) 0x7fce2c137df0 con 0x7fce2c13f710 2024-03-25T18:15:20.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.735+0000 7fce28ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:20.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.735+0000 7fce28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:20.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.735+0000 7fce28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:20.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.735+0000 7fce28ff9640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:15:20.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.735+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce18004eb0 con 0x7fce2c13f710 2024-03-25T18:15:20.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.757+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 906374475 0 0) 0x7fce18004eb0 con 0x7fce2c13f710 2024-03-25T18:15:20.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.757+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce2c137df0 con 0x7fce2c13f710 2024-03-25T18:15:20.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.757+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fce20001b00 con 0x7fce2c13f710 2024-03-25T18:15:20.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.757+0000 7fce28ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:20.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.757+0000 7fce28ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:20.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.757+0000 7fce28ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:20.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.757+0000 7fce28ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:20.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.765+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547584506 0 0) 0x7fce2c133930 con 0x7fce2c1430c0 2024-03-25T18:15:20.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.765+0000 7fce28ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:20.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.765+0000 7fce28ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:20.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.765+0000 7fce28ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:20.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.765+0000 7fce28ff9640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:15:20.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.765+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce18004eb0 con 0x7fce2c1430c0 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3001752153 0 0) 0x7fce2c137df0 con 0x7fce2c13f710 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 >> v1:172.21.15.162:6790/0 conn(0x7fce2c1430c0 legacy=0x7fce2c133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 >> v1:172.21.15.191:6789/0 conn(0x7fce2c0a98f0 legacy=0x7fce2c1392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 1 monclient: found mon.a 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c1367d0 con 0x7fce2c13f710 2024-03-25T18:15:20.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 5 monclient: authenticate success, global_id 4281 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 10 monclient: _renew_subs 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2c1369e0 con 0x7fce2c13f710 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 10 monclient: _renew_subs 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2c13b640 con 0x7fce2c13f710 2024-03-25T18:15:20.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.768+0000 7fce3236b640 1 librados: init done 2024-03-25T18:15:20.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.769+0000 7fce3236b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:20.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.769+0000 7fce3236b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:20.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.769+0000 7fce3236b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:20.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:20.769+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c0627b0 con 0x7fce2c13f710 2024-03-25T18:15:21.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.033+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce20004a90 con 0x7fce2c13f710 2024-03-25T18:15:21.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.033+0000 7fce28ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.033+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fce20004da0 con 0x7fce2c13f710 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.033+0000 7fce28ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.033+0000 7fce28ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.033+0000 7fce28ff9640 10 monclient: dump: 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:21.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.033+0000 7fce2a7fc640 4 set_mon_vals no callback set 2024-03-25T18:15:21.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.034+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fce2001ad70 con 0x7fce2c13f710 2024-03-25T18:15:21.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.034+0000 7fce2bfff640 10 monclient: get_auth_request con 0x7fce1806dd90 auth_method 0 2024-03-25T18:15:21.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.035+0000 7fce28ff9640 10 client.4281.objecter ms_handle_connect 0x7fce1806dd90 2024-03-25T18:15:21.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.039+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fce20054800 con 0x7fce2c13f710 2024-03-25T18:15:21.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.039+0000 7fce28ff9640 10 client.4281.objecter ms_dispatch 0x7fce2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:21.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.039+0000 7fce28ff9640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:21.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.039+0000 7fce28ff9640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:21.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.039+0000 7fce28ff9640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-25T18:15:21.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.039+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce20050310 con 0x7fce2c13f710 2024-03-25T18:15:21.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.039+0000 7fce28ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:21.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.039+0000 7fce28ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:21.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.256+0000 7fce3236b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:21.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.256+0000 7fce3236b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:21.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.256+0000 7fce3236b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:21.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.256+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce2c0008d0 con 0x7fce2c13f710 2024-03-25T18:15:21.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.290+0000 7fce28ff9640 1 -- 172.21.15.162:0/1964780225 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fce20052240 con 0x7fce2c13f710 2024-03-25T18:15:21.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.290+0000 7fce28ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:21.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.290+0000 7fce28ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:21.291 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:15:21.291 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:21.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.292+0000 7fce3236b640 10 librados: watch_flush enter 2024-03-25T18:15:21.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.292+0000 7fce3236b640 10 librados: watch_flush exit 2024-03-25T18:15:21.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-25T18:15:21.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 10 client.4281.objecter successfully canceled tick 2024-03-25T18:15:21.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 >> v1:172.21.15.191:6800/1552703795 conn(0x7fce1806dd90 legacy=0x7fce18070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 10 monclient: shutdown 2024-03-25T18:15:21.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:21.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 >> v1:172.21.15.162:6789/0 conn(0x7fce2c13f710 legacy=0x7fce2c1376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 shutdown_connections 2024-03-25T18:15:21.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 >> 172.21.15.162:0/1964780225 conn(0x7fce2c0b7ea0 msgr2=0x7fce2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:21.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 shutdown_connections 2024-03-25T18:15:21.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 1 -- 172.21.15.162:0/1964780225 wait complete. 2024-03-25T18:15:21.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:21.293+0000 7fce3236b640 1 librados: shutdown 2024-03-25T18:15:21.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:22.091 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:34.319 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:22.103 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:34.331 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:22.153 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:34.381 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:22.154 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:34.371 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:22.161 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:34.377 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:22.164 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:34.392 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:22.172 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:34.400 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:22.204 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:34.420 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:22.215 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:34.442 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.113 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:28.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.231+0000 7fc3622c9640 10 monclient: get_monmap_and_config 2024-03-25T18:15:28.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.231+0000 7fc3622c9640 10 monclient: build_initial_monmap 2024-03-25T18:15:28.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.231+0000 7fc3622c9640 10 monclient: monmap: 2024-03-25T18:15:28.232 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:28.232 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:28.232 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:15:28.232261+0000 2024-03-25T18:15:28.233 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:15:28.232261+0000 2024-03-25T18:15:28.233 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:28.233 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:28.233 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:28.233 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:15:28.233 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:15:28.233 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:28.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 1 Processor -- start 2024-03-25T18:15:28.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 1 -- start start 2024-03-25T18:15:28.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 10 monclient: init 2024-03-25T18:15:28.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:28.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:28.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc35c0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:28.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc35c0b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:28.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc35c14be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:28.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c0af940 con 0x7fc35c14be90 2024-03-25T18:15:28.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.232+0000 7fc3622c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c0b0a70 con 0x7fc35c0b4d60 2024-03-25T18:15:28.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.233+0000 7fc3622c9640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c0b0d50 con 0x7fc35c0b1730 2024-03-25T18:15:28.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.233+0000 7fc3622c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:28.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.233+0000 7fc3622c9640 10 monclient(hunting): authenticate will time out at 7627.506740s 2024-03-25T18:15:28.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.241+0000 7fc360866640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc35c14be90 0x7fc35c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:45928/0 (socket says 172.21.15.162:45928) 2024-03-25T18:15:28.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.241+0000 7fc360866640 1 -- 172.21.15.162:0/106146156 learned_addr learned my addr 172.21.15.162:0/106146156 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:28.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495799237 0 0) 0x7fc35c0b0d50 con 0x7fc35c0b1730 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc344003520 con 0x7fc35c0b1730 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584089995 0 0) 0x7fc35c0b0a70 con 0x7fc35c0b4d60 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:15:28.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.251+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c0b0d50 con 0x7fc35c0b4d60 2024-03-25T18:15:28.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.255+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076468859 0 0) 0x7fc35c0af940 con 0x7fc35c14be90 2024-03-25T18:15:28.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.255+0000 7fc35affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.255+0000 7fc35affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.255+0000 7fc35affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.255+0000 7fc35affd640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:15:28.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.255+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c0b0a70 con 0x7fc35c14be90 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1634848770 0 0) 0x7fc344003520 con 0x7fc35c0b1730 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 >> v1:172.21.15.191:6789/0 conn(0x7fc35c0b4d60 legacy=0x7fc35c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 >> v1:172.21.15.162:6789/0 conn(0x7fc35c14be90 legacy=0x7fc35c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 1 monclient: found mon.noname-b 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c0b1030 con 0x7fc35c0b1730 2024-03-25T18:15:28.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 10 monclient: _finish_auth 0 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc350001690 con 0x7fc35c0b1730 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc3622c9640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc3622c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc35affd640 10 monclient: dump: 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:28.268 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.269 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.269 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.269 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:28.269 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:28.269 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:28.269 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:28.269 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:28.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.265+0000 7fc3622c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.275+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3500017f0 con 0x7fc35c0b1730 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.275+0000 7fc35affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.275+0000 7fc35affd640 1 -- 172.21.15.162:0/106146156 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc350001c00 con 0x7fc35c0b1730 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.275+0000 7fc3622c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.275+0000 7fc3622c9640 4 set_mon_vals no callback set 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.275+0000 7fc35affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.275+0000 7fc35affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.275+0000 7fc35affd640 10 monclient: dump: 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:28.277 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.278 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.278 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.278 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:28.278 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:28.278 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:28.278 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:28.278 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.276+0000 7fc3622c9640 10 monclient: shutdown 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.276+0000 7fc3622c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.276+0000 7fc3622c9640 1 -- 172.21.15.162:0/106146156 >> v1:172.21.15.162:6790/0 conn(0x7fc35c0b1730 legacy=0x7fc35c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.276+0000 7fc3622c9640 1 -- 172.21.15.162:0/106146156 shutdown_connections 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.276+0000 7fc3622c9640 1 -- 172.21.15.162:0/106146156 >> 172.21.15.162:0/106146156 conn(0x7fc35c0617a0 msgr2=0x7fc35c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.276+0000 7fc3622c9640 1 -- 172.21.15.162:0/106146156 shutdown_connections 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.276+0000 7fc3622c9640 1 -- 172.21.15.162:0/106146156 wait complete. 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.277+0000 7fc3622c9640 10 monclient: build_initial_monmap 2024-03-25T18:15:28.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.277+0000 7fc3622c9640 10 monclient: monmap: 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:28.282 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.277+0000 7fc3622c9640 1 librados: starting msgr at 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.277+0000 7fc3622c9640 1 librados: starting objecter 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 1 Processor -- start 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 1 -- start start 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 1 librados: setting wanted keys 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 1 librados: calling monclient init 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 10 monclient: init 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:28.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 10 monclient(hunting): picked mon.noname-c con 0x7fc35c0b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:28.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 10 monclient(hunting): picked mon.noname-b con 0x7fc35c0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:28.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 10 monclient(hunting): picked mon.noname-a con 0x7fc35c14be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:28.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c1404d0 con 0x7fc35c14be90 2024-03-25T18:15:28.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c1daeb0 con 0x7fc35c0b4d60 2024-03-25T18:15:28.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c1380e0 con 0x7fc35c0b1730 2024-03-25T18:15:28.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:28.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.278+0000 7fc3622c9640 10 monclient(hunting): authenticate will time out at 7627.552620s 2024-03-25T18:15:28.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.290+0000 7fc35b7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc35c0b4d60 0x7fc35c1db3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:47376/0 (socket says 172.21.15.162:47376) 2024-03-25T18:15:28.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.290+0000 7fc35b7fe640 1 -- 172.21.15.162:0/3443911711 learned_addr learned my addr 172.21.15.162:0/3443911711 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:28.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.291+0000 7fc358ff9640 10 client.?.objecter ms_handle_connect 0x7fc35c0b4d60 2024-03-25T18:15:28.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.291+0000 7fc358ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:28.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.292+0000 7fc358ff9640 10 client.?.objecter ms_handle_connect 0x7fc35c0b1730 2024-03-25T18:15:28.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.292+0000 7fc358ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:28.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.297+0000 7fc358ff9640 10 client.?.objecter ms_handle_connect 0x7fc35c14be90 2024-03-25T18:15:28.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.297+0000 7fc358ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:28.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.300+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021742602 0 0) 0x7fc35c1380e0 con 0x7fc35c0b1730 2024-03-25T18:15:28.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.300+0000 7fc358ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.300+0000 7fc358ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.300+0000 7fc358ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.300+0000 7fc358ff9640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:15:28.302 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.300+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc348004eb0 con 0x7fc35c0b1730 2024-03-25T18:15:28.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.306+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109412121 0 0) 0x7fc35c1404d0 con 0x7fc35c14be90 2024-03-25T18:15:28.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:15:28.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c1380e0 con 0x7fc35c14be90 2024-03-25T18:15:28.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222393156 0 0) 0x7fc35c1daeb0 con 0x7fc35c0b4d60 2024-03-25T18:15:28.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:15:28.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.307+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35c1404d0 con 0x7fc35c0b4d60 2024-03-25T18:15:28.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.318+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1826616673 0 0) 0x7fc348004eb0 con 0x7fc35c0b1730 2024-03-25T18:15:28.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.318+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc35c1daeb0 con 0x7fc35c0b1730 2024-03-25T18:15:28.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.318+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc34c001970 con 0x7fc35c0b1730 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.318+0000 7fc358ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.318+0000 7fc358ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.318+0000 7fc358ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:28.320 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:28.321 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:28.321 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:28.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.318+0000 7fc358ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.320+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 335479562 0 0) 0x7fc35c1380e0 con 0x7fc35c14be90 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.320+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc348004eb0 con 0x7fc35c14be90 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.320+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc354001b50 con 0x7fc35c14be90 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.320+0000 7fc358ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.320+0000 7fc358ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.320+0000 7fc358ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:28.322 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.323 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.323 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.323 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:28.323 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:28.323 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:28.323 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:28.323 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:28.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4082025501 0 0) 0x7fc35c1daeb0 con 0x7fc35c0b1730 2024-03-25T18:15:28.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 >> v1:172.21.15.191:6789/0 conn(0x7fc35c0b4d60 legacy=0x7fc35c1db3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 >> v1:172.21.15.162:6789/0 conn(0x7fc35c14be90 legacy=0x7fc35c130de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:28.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 1 monclient: found mon.c 2024-03-25T18:15:28.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:28.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:28.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c1dbae0 con 0x7fc35c0b1730 2024-03-25T18:15:28.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:28.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:28.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc358ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:28.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc3622c9640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:15:28.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc3622c9640 10 monclient: _renew_subs 2024-03-25T18:15:28.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc3622c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.325+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c137310 con 0x7fc35c0b1730 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.326+0000 7fc3622c9640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.326+0000 7fc3622c9640 10 monclient: _renew_subs 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.326+0000 7fc3622c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.326+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c143d40 con 0x7fc35c0b1730 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.326+0000 7fc3622c9640 1 librados: init done 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.327+0000 7fc3622c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.327+0000 7fc3622c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.327+0000 7fc3622c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:28.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.327+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c1db040 con 0x7fc35c0b1730 2024-03-25T18:15:28.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.623+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc34c0035f0 con 0x7fc35c0b1730 2024-03-25T18:15:28.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.623+0000 7fc358ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:28.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.623+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc34c003900 con 0x7fc35c0b1730 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.623+0000 7fc358ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.623+0000 7fc358ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.623+0000 7fc358ff9640 10 monclient: dump: 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:28.626 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:28.627 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:28.627 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:28.627 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:28.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.623+0000 7fc35a7fc640 4 set_mon_vals no callback set 2024-03-25T18:15:28.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.624+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fc34c019910 con 0x7fc35c0b1730 2024-03-25T18:15:28.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.625+0000 7fc35b7fe640 10 monclient: get_auth_request con 0x7fc34806df20 auth_method 0 2024-03-25T18:15:28.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.625+0000 7fc358ff9640 10 client.4277.objecter ms_handle_connect 0x7fc34806df20 2024-03-25T18:15:28.632 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.630+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fc34c053640 con 0x7fc35c0b1730 2024-03-25T18:15:28.632 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.630+0000 7fc358ff9640 10 client.4277.objecter ms_dispatch 0x7fc35c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:28.632 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.630+0000 7fc358ff9640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:28.632 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.630+0000 7fc358ff9640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:28.632 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.630+0000 7fc358ff9640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T18:15:28.632 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.630+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc34c04ef80 con 0x7fc35c0b1730 2024-03-25T18:15:28.633 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.630+0000 7fc358ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:28.633 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.630+0000 7fc358ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:28.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.870+0000 7fc3622c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:28.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.870+0000 7fc3622c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:28.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.870+0000 7fc3622c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:15:28.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.870+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc35c1daca0 con 0x7fc35c0b1730 2024-03-25T18:15:28.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.919+0000 7fc358ff9640 1 -- 172.21.15.162:0/3443911711 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fc34c051080 con 0x7fc35c0b1730 2024-03-25T18:15:28.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.919+0000 7fc358ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:28.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.919+0000 7fc358ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:28.920 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:15:28.921 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:28.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.921+0000 7fc3622c9640 10 librados: watch_flush enter 2024-03-25T18:15:28.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.921+0000 7fc3622c9640 10 librados: watch_flush exit 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.921+0000 7fc3622c9640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.921+0000 7fc3622c9640 10 client.4277.objecter successfully canceled tick 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.921+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 >> v1:172.21.15.191:6800/1552703795 conn(0x7fc34806df20 legacy=0x7fc3480703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.922+0000 7fc3622c9640 10 monclient: shutdown 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.922+0000 7fc3622c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.922+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 >> v1:172.21.15.162:6790/0 conn(0x7fc35c0b1730 legacy=0x7fc35c1379d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.922+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 shutdown_connections 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.922+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 >> 172.21.15.162:0/3443911711 conn(0x7fc35c0617a0 msgr2=0x7fc35c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.922+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 shutdown_connections 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.922+0000 7fc3622c9640 1 -- 172.21.15.162:0/3443911711 wait complete. 2024-03-25T18:15:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:28.922+0000 7fc3622c9640 1 librados: shutdown 2024-03-25T18:15:28.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:35.742 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.855+0000 7f9657e72640 10 monclient: get_monmap_and_config 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.855+0000 7f9657e72640 10 monclient: build_initial_monmap 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.855+0000 7f9657e72640 10 monclient: monmap: 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:15:35.856203+0000 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:15:35.856203+0000 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:35.856 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:35.857 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:15:35.857 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:15:35.857 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.855+0000 7f9657e72640 1 Processor -- start 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 1 -- start start 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 10 monclient: init 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 10 monclient(hunting): picked mon.noname-b con 0x7f96500a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 10 monclient(hunting): picked mon.noname-a con 0x7f965013f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 10 monclient(hunting): picked mon.noname-c con 0x7f96501430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9650142520 con 0x7f965013f710 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96500b28a0 con 0x7f96501430c0 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96500b2a30 con 0x7f96500a98f0 2024-03-25T18:15:35.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:35.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.856+0000 7f9657e72640 10 monclient(hunting): authenticate will time out at 7635.130604s 2024-03-25T18:15:35.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.863+0000 7f965640f640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f96501430c0 0x7f965014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:43230/0 (socket says 172.21.15.162:43230) 2024-03-25T18:15:35.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.863+0000 7f965640f640 1 -- 172.21.15.162:0/2629090760 learned_addr learned my addr 172.21.15.162:0/2629090760 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634425736 0 0) 0x7f96500b28a0 con 0x7f96501430c0 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 10 monclient(hunting): _negotiate 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c003520 con 0x7f96501430c0 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251451906 0 0) 0x7f9650142520 con 0x7f965013f710 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 10 monclient(hunting): _negotiate 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:35.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:15:35.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.876+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96500b28a0 con 0x7f965013f710 2024-03-25T18:15:35.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.888+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3450463872 0 0) 0x7f96500b28a0 con 0x7f965013f710 2024-03-25T18:15:35.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 >> v1:172.21.15.162:6790/0 conn(0x7f96500a98f0 legacy=0x7f96500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 >> v1:172.21.15.191:6789/0 conn(0x7f96501430c0 legacy=0x7f965014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 10 monclient: _finish_hunting 0 2024-03-25T18:15:35.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 1 monclient: found mon.noname-a 2024-03-25T18:15:35.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:35.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:15:35.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96500b44c0 con 0x7f965013f710 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 10 monclient: _finish_auth 0 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 10 monclient: _check_auth_tickets 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9640001670 con 0x7f965013f710 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9654c0c640 10 monclient: dump: 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:35.891 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:35.892 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:35.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9657e72640 5 monclient: authenticate success, global_id 4290 2024-03-25T18:15:35.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.889+0000 7f9657e72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:35.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.897+0000 7f9654c0c640 1 -- 172.21.15.162:0/2629090760 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96400017d0 con 0x7f965013f710 2024-03-25T18:15:35.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.897+0000 7f9654c0c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:35.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.897+0000 7f9657e72640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:35.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.897+0000 7f9657e72640 4 set_mon_vals no callback set 2024-03-25T18:15:35.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 10 monclient: shutdown 2024-03-25T18:15:35.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:35.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 1 -- 172.21.15.162:0/2629090760 >> v1:172.21.15.162:6789/0 conn(0x7f965013f710 legacy=0x7f96501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 1 -- 172.21.15.162:0/2629090760 shutdown_connections 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 1 -- 172.21.15.162:0/2629090760 >> 172.21.15.162:0/2629090760 conn(0x7f96500b7ea0 msgr2=0x7f96500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 1 -- 172.21.15.162:0/2629090760 shutdown_connections 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 1 -- 172.21.15.162:0/2629090760 wait complete. 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 10 monclient: build_initial_monmap 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.898+0000 7f9657e72640 10 monclient: monmap: 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:15:35.900 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:35.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.899+0000 7f9657e72640 1 librados: starting msgr at 2024-03-25T18:15:35.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.899+0000 7f9657e72640 1 librados: starting objecter 2024-03-25T18:15:35.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.899+0000 7f9657e72640 1 Processor -- start 2024-03-25T18:15:35.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.899+0000 7f9657e72640 1 -- start start 2024-03-25T18:15:35.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.899+0000 7f9657e72640 1 librados: setting wanted keys 2024-03-25T18:15:35.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.899+0000 7f9657e72640 1 librados: calling monclient init 2024-03-25T18:15:35.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.899+0000 7f9657e72640 10 monclient: init 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 10 monclient(hunting): picked mon.noname-c con 0x7f96500a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 10 monclient(hunting): picked mon.noname-b con 0x7f965013f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 10 monclient(hunting): picked mon.noname-a con 0x7f96501430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96500b7cc0 con 0x7f96501430c0 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965013b0a0 con 0x7f965013f710 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96500b6e70 con 0x7f96500a98f0 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:35.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.900+0000 7f9657e72640 10 monclient(hunting): authenticate will time out at 7635.173986s 2024-03-25T18:15:35.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.902+0000 7f965540d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f965013f710 0x7f965012d0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:43256/0 (socket says 172.21.15.162:43256) 2024-03-25T18:15:35.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.902+0000 7f965540d640 1 -- 172.21.15.162:0/2649418338 learned_addr learned my addr 172.21.15.162:0/2649418338 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:35.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.904+0000 7f96467fc640 10 client.?.objecter ms_handle_connect 0x7f965013f710 2024-03-25T18:15:35.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.904+0000 7f96467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:35.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.909+0000 7f96467fc640 10 client.?.objecter ms_handle_connect 0x7f96501430c0 2024-03-25T18:15:35.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.909+0000 7f96467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:35.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.916+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99233135 0 0) 0x7f96500b7cc0 con 0x7f96501430c0 2024-03-25T18:15:35.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.916+0000 7f96467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:35.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.916+0000 7f96467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:35.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.916+0000 7f96467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:35.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.916+0000 7f96467fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:15:35.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.916+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9634004eb0 con 0x7f96501430c0 2024-03-25T18:15:35.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.919+0000 7f96467fc640 10 client.?.objecter ms_handle_connect 0x7f96500a98f0 2024-03-25T18:15:35.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.919+0000 7f96467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:35.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.922+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325257469 0 0) 0x7f965013b0a0 con 0x7f965013f710 2024-03-25T18:15:35.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.922+0000 7f96467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:35.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.922+0000 7f96467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:35.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.922+0000 7f96467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:35.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.922+0000 7f96467fc640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:15:35.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.922+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96500b7cc0 con 0x7f965013f710 2024-03-25T18:15:35.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.925+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3929435779 0 0) 0x7f9634004eb0 con 0x7f96501430c0 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.925+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f965013b0a0 con 0x7f96501430c0 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.925+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9648001b90 con 0x7f96501430c0 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.925+0000 7f96467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.925+0000 7f96467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.925+0000 7f96467fc640 10 monclient(hunting): dump: 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:35.926 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:35.927 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:35.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.925+0000 7f96467fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:35.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.934+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2967540606 0 0) 0x7f96500b7cc0 con 0x7f965013f710 2024-03-25T18:15:35.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.935+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9634004eb0 con 0x7f965013f710 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.935+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9640001650 con 0x7f965013f710 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.935+0000 7f96467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.935+0000 7f96467fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.935+0000 7f96467fc640 10 monclient(hunting): dump: 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:35.936 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 143709476 0 0) 0x7f965013b0a0 con 0x7f96501430c0 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 >> v1:172.21.15.162:6790/0 conn(0x7f96500a98f0 legacy=0x7f9650139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 >> v1:172.21.15.191:6789/0 conn(0x7f965013f710 legacy=0x7f965012d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 1 monclient: found mon.a 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96500b77a0 con 0x7f96501430c0 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f96467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:35.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f9657e72640 5 monclient: authenticate success, global_id 4293 2024-03-25T18:15:35.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f9657e72640 10 monclient: _renew_subs 2024-03-25T18:15:35.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f9657e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:35.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965012f470 con 0x7f96501430c0 2024-03-25T18:15:35.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.942+0000 7f9657e72640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:35.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.943+0000 7f9657e72640 10 monclient: _renew_subs 2024-03-25T18:15:35.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.943+0000 7f9657e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:35.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.943+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f965012d7b0 con 0x7f96501430c0 2024-03-25T18:15:35.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.943+0000 7f9657e72640 1 librados: init done 2024-03-25T18:15:35.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.943+0000 7f9657e72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:35.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.943+0000 7f9657e72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:35.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.943+0000 7f9657e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:35.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:35.943+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96500b3ef0 con 0x7f96501430c0 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.274+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9648001fe0 con 0x7f96501430c0 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.274+0000 7f96467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.274+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9648004d00 con 0x7f96501430c0 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.274+0000 7f96467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.274+0000 7f96467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.274+0000 7f96467fc640 10 monclient: dump: 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:36.275 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:36.276 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:36.276 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:36.276 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:36.276 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:36.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.274+0000 7f9647fff640 4 set_mon_vals no callback set 2024-03-25T18:15:36.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.274+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f9648004f20 con 0x7f96501430c0 2024-03-25T18:15:36.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.275+0000 7f9655c0e640 10 monclient: get_auth_request con 0x7f963406dd90 auth_method 0 2024-03-25T18:15:36.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.275+0000 7f96467fc640 10 client.4293.objecter ms_handle_connect 0x7f963406dd90 2024-03-25T18:15:36.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.282+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f9648054a30 con 0x7f96501430c0 2024-03-25T18:15:36.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.282+0000 7f96467fc640 10 client.4293.objecter ms_dispatch 0x7f9650000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:36.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.282+0000 7f96467fc640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:36.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.282+0000 7f96467fc640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:36.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.282+0000 7f96467fc640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T18:15:36.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.282+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9648050540 con 0x7f96501430c0 2024-03-25T18:15:36.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.282+0000 7f96467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:36.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.282+0000 7f96467fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:36.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.531+0000 7f9657e72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:36.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.531+0000 7f9657e72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:36.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.531+0000 7f9657e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:36.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.531+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96500627b0 con 0x7f96501430c0 2024-03-25T18:15:36.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.617+0000 7f96467fc640 1 -- 172.21.15.162:0/2649418338 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f9648052470 con 0x7f96501430c0 2024-03-25T18:15:36.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.617+0000 7f96467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:36.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.617+0000 7f96467fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:36.618 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:15:36.619 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 10 librados: watch_flush enter 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 10 librados: watch_flush exit 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 10 client.4293.objecter successfully canceled tick 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 >> v1:172.21.15.191:6800/1552703795 conn(0x7f963406dd90 legacy=0x7f9634070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 10 monclient: shutdown 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 >> v1:172.21.15.162:6789/0 conn(0x7f96501430c0 legacy=0x7f96501d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 shutdown_connections 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.619+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 >> 172.21.15.162:0/2649418338 conn(0x7f96500b7ea0 msgr2=0x7f9650074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:36.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.620+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 shutdown_connections 2024-03-25T18:15:36.622 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.620+0000 7f9657e72640 1 -- 172.21.15.162:0/2649418338 wait complete. 2024-03-25T18:15:36.622 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:36.620+0000 7f9657e72640 1 librados: shutdown 2024-03-25T18:15:36.630 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:42.179 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:54.407 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:42.187 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:54.415 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:42.225 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:01:54.453 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:42.231 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:54.459 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:42.240 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:54.468 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:42.246 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:54.462 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:42.262 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:54.478 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:42.269 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:01:54.497 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:42.279 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:01:54.496 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:43.437 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.551+0000 7fc53c886640 10 monclient: get_monmap_and_config 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.551+0000 7fc53c886640 10 monclient: build_initial_monmap 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.551+0000 7fc53c886640 10 monclient: monmap: 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:15:43.552222+0000 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:15:43.552222+0000 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:43.552 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:43.553 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:43.553 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:15:43.553 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:15:43.553 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.551+0000 7fc53c886640 1 Processor -- start 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 1 -- start start 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 10 monclient: init 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 10 monclient(hunting): picked mon.noname-c con 0x7fc5380b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 10 monclient(hunting): picked mon.noname-b con 0x7fc5380b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 10 monclient(hunting): picked mon.noname-a con 0x7fc53814be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:43.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5380af940 con 0x7fc53814be90 2024-03-25T18:15:43.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5380b0a70 con 0x7fc5380b4d60 2024-03-25T18:15:43.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5380b0d50 con 0x7fc5380b1730 2024-03-25T18:15:43.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:43.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.552+0000 7fc53c886640 10 monclient(hunting): authenticate will time out at 7642.826439s 2024-03-25T18:15:43.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.564+0000 7fc52ffff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fc5380b1730 0x7fc5380b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:45162/0 (socket says 172.21.15.162:45162) 2024-03-25T18:15:43.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.564+0000 7fc52ffff640 1 -- 172.21.15.162:0/1880966052 learned_addr learned my addr 172.21.15.162:0/1880966052 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:43.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.577+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137721222 0 0) 0x7fc5380b0a70 con 0x7fc5380b4d60 2024-03-25T18:15:43.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.577+0000 7fc535d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:43.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.577+0000 7fc535d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:43.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.577+0000 7fc535d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:43.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.577+0000 7fc535d9b640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:15:43.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.577+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc524003520 con 0x7fc5380b4d60 2024-03-25T18:15:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.587+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434571876 0 0) 0x7fc5380af940 con 0x7fc53814be90 2024-03-25T18:15:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.587+0000 7fc535d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.587+0000 7fc535d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.587+0000 7fc535d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.587+0000 7fc535d9b640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:15:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.587+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5380b0a70 con 0x7fc53814be90 2024-03-25T18:15:43.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3865630593 0 0) 0x7fc524003520 con 0x7fc5380b4d60 2024-03-25T18:15:43.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 >> v1:172.21.15.162:6790/0 conn(0x7fc5380b1730 legacy=0x7fc5380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:43.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 >> v1:172.21.15.162:6789/0 conn(0x7fc53814be90 legacy=0x7fc53814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 1 monclient: found mon.noname-c 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5380b1030 con 0x7fc5380b4d60 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc528001690 con 0x7fc5380b4d60 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc535d9b640 10 monclient: dump: 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:43.591 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc53c886640 5 monclient: authenticate success, global_id 4282 2024-03-25T18:15:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.589+0000 7fc53c886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.595+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5280017f0 con 0x7fc5380b4d60 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.595+0000 7fc535d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.595+0000 7fc535d9b640 1 -- 172.21.15.162:0/1880966052 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc528001c00 con 0x7fc5380b4d60 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.595+0000 7fc53c886640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.595+0000 7fc53c886640 4 set_mon_vals no callback set 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.595+0000 7fc535d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.595+0000 7fc535d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.595+0000 7fc535d9b640 10 monclient: dump: 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:43.596 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.597 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.597 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:43.597 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:43.597 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:43.597 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:43.597 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:43.597 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:43.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.596+0000 7fc53c886640 10 monclient: shutdown 2024-03-25T18:15:43.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.596+0000 7fc53c886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:43.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.596+0000 7fc53c886640 1 -- 172.21.15.162:0/1880966052 >> v1:172.21.15.191:6789/0 conn(0x7fc5380b4d60 legacy=0x7fc53813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:43.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.596+0000 7fc53c886640 1 -- 172.21.15.162:0/1880966052 shutdown_connections 2024-03-25T18:15:43.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.596+0000 7fc53c886640 1 -- 172.21.15.162:0/1880966052 >> 172.21.15.162:0/1880966052 conn(0x7fc5380617a0 msgr2=0x7fc538061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:43.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.596+0000 7fc53c886640 1 -- 172.21.15.162:0/1880966052 shutdown_connections 2024-03-25T18:15:43.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.596+0000 7fc53c886640 1 -- 172.21.15.162:0/1880966052 wait complete. 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 10 monclient: build_initial_monmap 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 10 monclient: monmap: 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 1 librados: starting msgr at 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 1 librados: starting objecter 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 1 Processor -- start 2024-03-25T18:15:43.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 1 -- start start 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 1 librados: setting wanted keys 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 1 librados: calling monclient init 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.597+0000 7fc53c886640 10 monclient: init 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 10 monclient(hunting): picked mon.noname-a con 0x7fc5380b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 10 monclient(hunting): picked mon.noname-b con 0x7fc5380b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 10 monclient(hunting): picked mon.noname-c con 0x7fc53814be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc538141ce0 con 0x7fc5380b1730 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc538142610 con 0x7fc5380b4d60 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5381dae80 con 0x7fc53814be90 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:43.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.598+0000 7fc53c886640 10 monclient(hunting): authenticate will time out at 7642.871831s 2024-03-25T18:15:43.613 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.613+0000 7fc53659c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc5380b1730 0x7fc538142b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:44002/0 (socket says 172.21.15.162:44002) 2024-03-25T18:15:43.613 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.613+0000 7fc53659c640 1 -- 172.21.15.162:0/2473947767 learned_addr learned my addr 172.21.15.162:0/2473947767 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:43.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.617+0000 7fc52effd640 10 client.?.objecter ms_handle_connect 0x7fc5380b1730 2024-03-25T18:15:43.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.617+0000 7fc52effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:43.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.618+0000 7fc52effd640 10 client.?.objecter ms_handle_connect 0x7fc5380b4d60 2024-03-25T18:15:43.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.618+0000 7fc52effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:43.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.623+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756071921 0 0) 0x7fc538142610 con 0x7fc5380b4d60 2024-03-25T18:15:43.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.623+0000 7fc52effd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:43.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.623+0000 7fc52effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:43.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.623+0000 7fc52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:43.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.623+0000 7fc52effd640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:15:43.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.623+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc518004eb0 con 0x7fc5380b4d60 2024-03-25T18:15:43.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.626+0000 7fc52effd640 10 client.?.objecter ms_handle_connect 0x7fc53814be90 2024-03-25T18:15:43.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.626+0000 7fc52effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:43.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.628+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2008703754 0 0) 0x7fc538141ce0 con 0x7fc5380b1730 2024-03-25T18:15:43.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.628+0000 7fc52effd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:43.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.628+0000 7fc52effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:43.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.628+0000 7fc52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:43.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.628+0000 7fc52effd640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:15:43.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.628+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5180051d0 con 0x7fc5380b1730 2024-03-25T18:15:43.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.640+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417656298 0 0) 0x7fc5381dae80 con 0x7fc53814be90 2024-03-25T18:15:43.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.640+0000 7fc52effd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:43.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.640+0000 7fc52effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:43.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.640+0000 7fc52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:43.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.640+0000 7fc52effd640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:15:43.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.640+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc538141ce0 con 0x7fc53814be90 2024-03-25T18:15:43.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.642+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339900565 0 0) 0x7fc5180051d0 con 0x7fc5380b1730 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5381dae80 con 0x7fc5380b1730 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc528001690 con 0x7fc5380b1730 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 10 monclient(hunting): dump: 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:43.644 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:43.645 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:43.645 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 847350902 0 0) 0x7fc518004eb0 con 0x7fc5380b4d60 2024-03-25T18:15:43.645 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5180051d0 con 0x7fc5380b4d60 2024-03-25T18:15:43.645 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc5200018a0 con 0x7fc5380b4d60 2024-03-25T18:15:43.645 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:43.645 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.643+0000 7fc52effd640 10 monclient(hunting): dump: 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:43.646 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.650+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 441136999 0 0) 0x7fc538141ce0 con 0x7fc53814be90 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.650+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc518004eb0 con 0x7fc53814be90 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.650+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc530001bd0 con 0x7fc53814be90 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.650+0000 7fc52effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.650+0000 7fc52effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.650+0000 7fc52effd640 10 monclient(hunting): dump: 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:43.651 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.652 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.652 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:43.652 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:43.652 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:43.652 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:43.652 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:43.652 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:43.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1096308452 0 0) 0x7fc5180051d0 con 0x7fc5380b4d60 2024-03-25T18:15:43.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 >> v1:172.21.15.162:6790/0 conn(0x7fc53814be90 legacy=0x7fc538140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:43.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 >> v1:172.21.15.162:6789/0 conn(0x7fc5380b1730 legacy=0x7fc538142b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:43.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:43.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 1 monclient: found mon.b 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5381db010 con 0x7fc5380b4d60 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 10 monclient: _finish_auth 0 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc52effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 10 monclient: _renew_subs 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc538143240 con 0x7fc5380b4d60 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 10 monclient: _renew_subs 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53813bd20 con 0x7fc5380b4d60 2024-03-25T18:15:43.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.655+0000 7fc53c886640 1 librados: init done 2024-03-25T18:15:43.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.656+0000 7fc53c886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:43.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.656+0000 7fc53c886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:43.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.656+0000 7fc53c886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:43.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.656+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5381427a0 con 0x7fc5380b4d60 2024-03-25T18:15:43.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.926+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc520001d30 con 0x7fc5380b4d60 2024-03-25T18:15:43.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.926+0000 7fc52effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:43.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.926+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc520004980 con 0x7fc5380b4d60 2024-03-25T18:15:43.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.926+0000 7fc52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.926+0000 7fc52effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.926+0000 7fc52effd640 10 monclient: dump: 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:43.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.926+0000 7fc53559a640 4 set_mon_vals no callback set 2024-03-25T18:15:43.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.927+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fc520004ba0 con 0x7fc5380b4d60 2024-03-25T18:15:43.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.928+0000 7fc53659c640 10 monclient: get_auth_request con 0x7fc51806e2b0 auth_method 0 2024-03-25T18:15:43.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.928+0000 7fc52effd640 10 client.4285.objecter ms_handle_connect 0x7fc51806e2b0 2024-03-25T18:15:43.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.932+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fc5200546d0 con 0x7fc5380b4d60 2024-03-25T18:15:43.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.932+0000 7fc52effd640 10 client.4285.objecter ms_dispatch 0x7fc538000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:43.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.932+0000 7fc52effd640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:43.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.932+0000 7fc52effd640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:43.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.932+0000 7fc52effd640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-25T18:15:43.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.932+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc520050010 con 0x7fc5380b4d60 2024-03-25T18:15:43.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.933+0000 7fc52effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:43.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:43.933+0000 7fc52effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:44.179 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.178+0000 7fc53c886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:44.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.178+0000 7fc53c886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:44.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.178+0000 7fc53c886640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:44.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.178+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5380627b0 con 0x7fc5380b4d60 2024-03-25T18:15:44.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.213+0000 7fc52effd640 1 -- 172.21.15.162:0/2473947767 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fc520052110 con 0x7fc5380b4d60 2024-03-25T18:15:44.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.213+0000 7fc52effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:44.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.213+0000 7fc52effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:44.215 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:15:44.215 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:44.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 10 librados: watch_flush enter 2024-03-25T18:15:44.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 10 librados: watch_flush exit 2024-03-25T18:15:44.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-25T18:15:44.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 10 client.4285.objecter successfully canceled tick 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 >> v1:172.21.15.191:6800/1552703795 conn(0x7fc51806e2b0 legacy=0x7fc518070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 10 monclient: shutdown 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 >> v1:172.21.15.191:6789/0 conn(0x7fc5380b4d60 legacy=0x7fc5381db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 shutdown_connections 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 >> 172.21.15.162:0/2473947767 conn(0x7fc5380617a0 msgr2=0x7fc5380b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 shutdown_connections 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 1 -- 172.21.15.162:0/2473947767 wait complete. 2024-03-25T18:15:44.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:44.216+0000 7fc53c886640 1 librados: shutdown 2024-03-25T18:15:44.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:51.035 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.154+0000 7f3e1cfd0640 10 monclient: get_monmap_and_config 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.154+0000 7f3e1cfd0640 10 monclient: build_initial_monmap 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.154+0000 7f3e1cfd0640 10 monclient: monmap: 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:15:51.155421+0000 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:15:51.155421+0000 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:51.155 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:51.156 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:51.156 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:15:51.156 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:15:51.156 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 1 Processor -- start 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 1 -- start start 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 10 monclient: init 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 10 monclient(hunting): picked mon.noname-c con 0x7f3e180b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 10 monclient(hunting): picked mon.noname-b con 0x7f3e180b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 10 monclient(hunting): picked mon.noname-a con 0x7f3e1814be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e180af940 con 0x7f3e1814be90 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e180b0a70 con 0x7f3e180b4d60 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e180b0d50 con 0x7f3e180b1730 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:51.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.155+0000 7f3e1cfd0640 10 monclient(hunting): authenticate will time out at 7650.429609s 2024-03-25T18:15:51.169 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.168+0000 7f3e1659c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3e180b4d60 0x7f3e1813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:37388/0 (socket says 172.21.15.162:37388) 2024-03-25T18:15:51.169 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.168+0000 7f3e1659c640 1 -- 172.21.15.162:0/2358596317 learned_addr learned my addr 172.21.15.162:0/2358596317 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:51.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.184+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779209522 0 0) 0x7f3e180b0a70 con 0x7f3e180b4d60 2024-03-25T18:15:51.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.184+0000 7f3e15d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:51.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.184+0000 7f3e15d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:51.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.184+0000 7f3e15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:51.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.184+0000 7f3e15d9b640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:15:51.185 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.184+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e04003520 con 0x7f3e180b4d60 2024-03-25T18:15:51.186 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.185+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337899355 0 0) 0x7f3e180af940 con 0x7f3e1814be90 2024-03-25T18:15:51.186 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.185+0000 7f3e15d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:51.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.185+0000 7f3e15d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:51.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.185+0000 7f3e15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:51.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.185+0000 7f3e15d9b640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:15:51.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.185+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e180b0a70 con 0x7f3e1814be90 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236031688 0 0) 0x7f3e180b0a70 con 0x7f3e1814be90 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 >> v1:172.21.15.162:6790/0 conn(0x7f3e180b1730 legacy=0x7f3e180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 >> v1:172.21.15.191:6789/0 conn(0x7f3e180b4d60 legacy=0x7f3e1813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 1 monclient: found mon.noname-a 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e180b1030 con 0x7f3e1814be90 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.192+0000 7f3e15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:51.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.193+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3e100019f0 con 0x7f3e1814be90 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.193+0000 7f3e1cfd0640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.193+0000 7f3e15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.193+0000 7f3e15d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.193+0000 7f3e15d9b640 10 monclient: dump: 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:51.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.193+0000 7f3e1cfd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:51.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.213+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e10001b50 con 0x7f3e1814be90 2024-03-25T18:15:51.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.213+0000 7f3e15d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:51.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.213+0000 7f3e15d9b640 1 -- 172.21.15.162:0/2358596317 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3e10001f20 con 0x7f3e1814be90 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.213+0000 7f3e1cfd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.213+0000 7f3e1cfd0640 4 set_mon_vals no callback set 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.213+0000 7f3e15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.213+0000 7f3e15d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.213+0000 7f3e15d9b640 10 monclient: dump: 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:51.215 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:51.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.214+0000 7f3e1cfd0640 10 monclient: shutdown 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.214+0000 7f3e1cfd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.214+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/2358596317 >> v1:172.21.15.162:6789/0 conn(0x7f3e1814be90 legacy=0x7f3e1814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.214+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/2358596317 shutdown_connections 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.214+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/2358596317 >> 172.21.15.162:0/2358596317 conn(0x7f3e180617a0 msgr2=0x7f3e18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.214+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/2358596317 shutdown_connections 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.214+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/2358596317 wait complete. 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 10 monclient: build_initial_monmap 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 10 monclient: monmap: 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:15:51.218 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 1 librados: starting msgr at 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 1 librados: starting objecter 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 1 Processor -- start 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 1 -- start start 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 1 librados: setting wanted keys 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 1 librados: calling monclient init 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.215+0000 7f3e1cfd0640 10 monclient: init 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 10 monclient(hunting): picked mon.noname-a con 0x7f3e180b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:51.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 10 monclient(hunting): picked mon.noname-b con 0x7f3e180b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:51.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 10 monclient(hunting): picked mon.noname-c con 0x7f3e1814be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:51.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e18138840 con 0x7f3e180b1730 2024-03-25T18:15:51.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e180aa660 con 0x7f3e180b4d60 2024-03-25T18:15:51.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e181d8420 con 0x7f3e1814be90 2024-03-25T18:15:51.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:51.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.216+0000 7f3e1cfd0640 10 monclient(hunting): authenticate will time out at 7650.490540s 2024-03-25T18:15:51.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.225+0000 7f3e0ffff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3e180b4d60 0x7f3e18137a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:37436/0 (socket says 172.21.15.162:37436) 2024-03-25T18:15:51.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.225+0000 7f3e0ffff640 1 -- 172.21.15.162:0/535230821 learned_addr learned my addr 172.21.15.162:0/535230821 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:51.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.227+0000 7f3e0effd640 10 client.?.objecter ms_handle_connect 0x7f3e180b4d60 2024-03-25T18:15:51.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.227+0000 7f3e0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:51.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.233+0000 7f3e0effd640 10 client.?.objecter ms_handle_connect 0x7f3e180b1730 2024-03-25T18:15:51.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.233+0000 7f3e0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:51.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.234+0000 7f3e0effd640 10 client.?.objecter ms_handle_connect 0x7f3e1814be90 2024-03-25T18:15:51.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.234+0000 7f3e0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:51.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.234+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880387505 0 0) 0x7f3e180aa660 con 0x7f3e180b4d60 2024-03-25T18:15:51.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.234+0000 7f3e0effd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:51.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.234+0000 7f3e0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:51.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.234+0000 7f3e0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:51.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.234+0000 7f3e0effd640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:15:51.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.234+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df8004eb0 con 0x7f3e180b4d60 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.244+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1369183237 0 0) 0x7f3df8004eb0 con 0x7f3e180b4d60 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.245+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e180aa660 con 0x7f3e180b4d60 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.245+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3e00001a30 con 0x7f3e180b4d60 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.245+0000 7f3e0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.245+0000 7f3e0effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.245+0000 7f3e0effd640 10 monclient(hunting): dump: 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:51.246 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:51.247 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:51.247 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:51.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.245+0000 7f3e0effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:51.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3093956738 0 0) 0x7f3e180aa660 con 0x7f3e180b4d60 2024-03-25T18:15:51.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 >> v1:172.21.15.162:6790/0 conn(0x7f3e1814be90 legacy=0x7f3e18138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 >> v1:172.21.15.162:6789/0 conn(0x7f3e180b1730 legacy=0x7f3e181d7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:51.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 1 monclient: found mon.b 2024-03-25T18:15:51.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e181d85b0 con 0x7f3e180b4d60 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 10 monclient: _finish_auth 0 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 10 monclient: _renew_subs 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e181426f0 con 0x7f3e180b4d60 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 10 monclient: _renew_subs 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e18140560 con 0x7f3e180b4d60 2024-03-25T18:15:51.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.250+0000 7f3e1cfd0640 1 librados: init done 2024-03-25T18:15:51.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.251+0000 7f3e1cfd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:51.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.251+0000 7f3e1cfd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:51.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.251+0000 7f3e1cfd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:51.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.251+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e180afbd0 con 0x7f3e180b4d60 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.528+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e00001cb0 con 0x7f3e180b4d60 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.528+0000 7f3e0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.528+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3e00003a60 con 0x7f3e180b4d60 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.528+0000 7f3e0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.528+0000 7f3e0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.528+0000 7f3e1559a640 4 set_mon_vals no callback set 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.528+0000 7f3e0effd640 10 monclient: dump: 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:51.529 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:51.530 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:51.530 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:51.530 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:51.530 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:51.530 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:51.530 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:51.530 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:51.530 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:51.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.529+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f3e00003c80 con 0x7f3e180b4d60 2024-03-25T18:15:51.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.529+0000 7f3e1659c640 10 monclient: get_auth_request con 0x7f3df806dc90 auth_method 0 2024-03-25T18:15:51.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.530+0000 7f3e0effd640 10 client.4291.objecter ms_handle_connect 0x7f3df806dc90 2024-03-25T18:15:51.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.534+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f3e00054910 con 0x7f3e180b4d60 2024-03-25T18:15:51.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.534+0000 7f3e0effd640 10 client.4291.objecter ms_dispatch 0x7f3e18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:51.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.534+0000 7f3e0effd640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:51.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.534+0000 7f3e0effd640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:51.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.535+0000 7f3e0effd640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T18:15:51.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.535+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e00050250 con 0x7f3e180b4d60 2024-03-25T18:15:51.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.535+0000 7f3e0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:51.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.535+0000 7f3e0effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:51.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.786+0000 7f3e1cfd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:51.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.786+0000 7f3e1cfd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:51.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.786+0000 7f3e1cfd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:15:51.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.786+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e180627b0 con 0x7f3e180b4d60 2024-03-25T18:15:51.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.808+0000 7f3e0effd640 1 -- 172.21.15.162:0/535230821 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f3e00052350 con 0x7f3e180b4d60 2024-03-25T18:15:51.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.808+0000 7f3e0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:51.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.808+0000 7f3e0effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:51.810 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:15:51.810 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 10 librados: watch_flush enter 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 10 librados: watch_flush exit 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 10 client.4291.objecter successfully canceled tick 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 >> v1:172.21.15.191:6800/1552703795 conn(0x7f3df806dc90 legacy=0x7f3df8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 10 monclient: shutdown 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 >> v1:172.21.15.191:6789/0 conn(0x7f3e180b4d60 legacy=0x7f3e18137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 shutdown_connections 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.811+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 >> 172.21.15.162:0/535230821 conn(0x7f3e180617a0 msgr2=0x7f3e180b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.812+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 shutdown_connections 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.812+0000 7f3e1cfd0640 1 -- 172.21.15.162:0/535230821 wait complete. 2024-03-25T18:15:51.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:51.812+0000 7f3e1cfd0640 1 librados: shutdown 2024-03-25T18:15:51.824 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:58.629 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:58.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.737+0000 7f17aad9d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:58.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.737+0000 7f17aad9d640 10 monclient: build_initial_monmap 2024-03-25T18:15:58.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.737+0000 7f17aad9d640 10 monclient: monmap: 2024-03-25T18:15:58.738 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:58.738 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:58.739 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:15:58.738847+0000 2024-03-25T18:15:58.739 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:15:58.738847+0000 2024-03-25T18:15:58.739 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:58.739 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:58.739 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:58.739 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:15:58.739 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:15:58.739 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:58.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.738+0000 7f17aad9d640 1 Processor -- start 2024-03-25T18:15:58.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.738+0000 7f17aad9d640 1 -- start start 2024-03-25T18:15:58.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.738+0000 7f17aad9d640 10 monclient: init 2024-03-25T18:15:58.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 10 monclient(hunting): picked mon.noname-b con 0x7f17a40b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 10 monclient(hunting): picked mon.noname-c con 0x7f17a40b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 10 monclient(hunting): picked mon.noname-a con 0x7f17a414be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a40af940 con 0x7f17a414be90 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a40b0a70 con 0x7f17a40b1730 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a40b0d50 con 0x7f17a40b4d60 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17aad9d640 10 monclient(hunting): authenticate will time out at 7658.012864s 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17a3fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f17a40b1730 0x7f17a40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:56644/0 (socket says 172.21.15.162:56644) 2024-03-25T18:15:58.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.739+0000 7f17a3fff640 1 -- 172.21.15.162:0/2384603723 learned_addr learned my addr 172.21.15.162:0/2384603723 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:58.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.797+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106424121 0 0) 0x7f17a40b0d50 con 0x7f17a40b4d60 2024-03-25T18:15:58.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.797+0000 7f17a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:58.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.797+0000 7f17a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:58.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.797+0000 7f17a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:58.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.797+0000 7f17a37fe640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:15:58.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.797+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1794003520 con 0x7f17a40b4d60 2024-03-25T18:15:58.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.801+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786423788 0 0) 0x7f17a40af940 con 0x7f17a414be90 2024-03-25T18:15:58.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.801+0000 7f17a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:58.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.801+0000 7f17a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:58.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.801+0000 7f17a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:58.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.801+0000 7f17a37fe640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:15:58.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.801+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17a40b0d50 con 0x7f17a414be90 2024-03-25T18:15:58.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3254048226 0 0) 0x7f17a40b0d50 con 0x7f17a414be90 2024-03-25T18:15:58.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 >> v1:172.21.15.162:6790/0 conn(0x7f17a40b4d60 legacy=0x7f17a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:58.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 >> v1:172.21.15.191:6789/0 conn(0x7f17a40b1730 legacy=0x7f17a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:58.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 1 monclient: found mon.noname-a 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a40b1030 con 0x7f17a414be90 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f179c0019f0 con 0x7f17a414be90 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17a37fe640 10 monclient: dump: 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:58.816 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:58.817 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:58.817 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:58.817 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:58.817 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:58.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17aad9d640 5 monclient: authenticate success, global_id 4308 2024-03-25T18:15:58.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.814+0000 7f17aad9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:58.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f179c001b50 con 0x7f17a414be90 2024-03-25T18:15:58.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:58.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17a37fe640 1 -- 172.21.15.162:0/2384603723 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f179c001f20 con 0x7f17a414be90 2024-03-25T18:15:58.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17aad9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:58.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17aad9d640 4 set_mon_vals no callback set 2024-03-25T18:15:58.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:58.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:58.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17a37fe640 10 monclient: dump: 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:58.851 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:58.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17aad9d640 10 monclient: shutdown 2024-03-25T18:15:58.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17aad9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:58.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17aad9d640 1 -- 172.21.15.162:0/2384603723 >> v1:172.21.15.162:6789/0 conn(0x7f17a414be90 legacy=0x7f17a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:58.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.849+0000 7f17aad9d640 1 -- 172.21.15.162:0/2384603723 shutdown_connections 2024-03-25T18:15:58.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.850+0000 7f17aad9d640 1 -- 172.21.15.162:0/2384603723 >> 172.21.15.162:0/2384603723 conn(0x7f17a40617a0 msgr2=0x7f17a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.850+0000 7f17aad9d640 1 -- 172.21.15.162:0/2384603723 shutdown_connections 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.850+0000 7f17aad9d640 1 -- 172.21.15.162:0/2384603723 wait complete. 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.850+0000 7f17aad9d640 10 monclient: build_initial_monmap 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.850+0000 7f17aad9d640 10 monclient: monmap: 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:15:58.855 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.851+0000 7f17aad9d640 1 librados: starting msgr at 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.851+0000 7f17aad9d640 1 librados: starting objecter 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.851+0000 7f17aad9d640 1 Processor -- start 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.851+0000 7f17aad9d640 1 -- start start 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.851+0000 7f17aad9d640 1 librados: setting wanted keys 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.851+0000 7f17aad9d640 1 librados: calling monclient init 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.851+0000 7f17aad9d640 10 monclient: init 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 10 monclient(hunting): picked mon.noname-b con 0x7f17a40b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 10 monclient(hunting): picked mon.noname-a con 0x7f17a40b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 10 monclient(hunting): picked mon.noname-c con 0x7f17a414be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:15:58.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a41421e0 con 0x7f17a40b4d60 2024-03-25T18:15:58.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a4132a00 con 0x7f17a40b1730 2024-03-25T18:15:58.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a4142f30 con 0x7f17a414be90 2024-03-25T18:15:58.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:58.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17aad9d640 10 monclient(hunting): authenticate will time out at 7658.126040s 2024-03-25T18:15:58.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17a933a640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f17a414be90 0x7f17a41322f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:36832/0 (socket says 172.21.15.162:36832) 2024-03-25T18:15:58.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.852+0000 7f17a933a640 1 -- 172.21.15.162:0/243164354 learned_addr learned my addr 172.21.15.162:0/243164354 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:15:58.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.853+0000 7f17a17fa640 10 client.?.objecter ms_handle_connect 0x7f17a414be90 2024-03-25T18:15:58.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.853+0000 7f17a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:58.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.862+0000 7f17a17fa640 10 client.?.objecter ms_handle_connect 0x7f17a40b1730 2024-03-25T18:15:58.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.862+0000 7f17a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:58.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.922+0000 7f17a17fa640 10 client.?.objecter ms_handle_connect 0x7f17a40b4d60 2024-03-25T18:15:58.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.922+0000 7f17a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:58.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.930+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424995015 0 0) 0x7f17a41421e0 con 0x7f17a40b4d60 2024-03-25T18:15:58.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.930+0000 7f17a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:58.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.930+0000 7f17a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:58.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.930+0000 7f17a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:58.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.930+0000 7f17a17fa640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:15:58.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.930+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1788004eb0 con 0x7f17a40b4d60 2024-03-25T18:15:58.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.949+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4261180763 0 0) 0x7f1788004eb0 con 0x7f17a40b4d60 2024-03-25T18:15:58.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.949+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17a41421e0 con 0x7f17a40b4d60 2024-03-25T18:15:58.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.949+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f17900018a0 con 0x7f17a40b4d60 2024-03-25T18:15:58.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.949+0000 7f17a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:58.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.949+0000 7f17a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:58.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.949+0000 7f17a17fa640 10 monclient(hunting): dump: 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:58.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.949+0000 7f17a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.955+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4118850758 0 0) 0x7f17a41421e0 con 0x7f17a40b4d60 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 >> v1:172.21.15.162:6790/0 conn(0x7f17a414be90 legacy=0x7f17a41322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 >> v1:172.21.15.191:6789/0 conn(0x7f17a40b1730 legacy=0x7f17a413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 1 monclient: found mon.a 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a4141cc0 con 0x7f17a40b4d60 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:58.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 5 monclient: authenticate success, global_id 4311 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 10 monclient: _renew_subs 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a4140380 con 0x7f17a40b4d60 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 10 monclient: _renew_subs 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a41418f0 con 0x7f17a40b4d60 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.956+0000 7f17aad9d640 1 librados: init done 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.957+0000 7f17aad9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.957+0000 7f17aad9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:58.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.957+0000 7f17aad9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:58.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:58.957+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a4141f00 con 0x7f17a40b4d60 2024-03-25T18:15:59.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.170+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17900047e0 con 0x7f17a40b4d60 2024-03-25T18:15:59.171 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.170+0000 7f17a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:59.171 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.170+0000 7f17a2ffd640 4 set_mon_vals no callback set 2024-03-25T18:15:59.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.245+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1790004a50 con 0x7f17a40b4d60 2024-03-25T18:15:59.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.245+0000 7f17a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:59.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.245+0000 7f17a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:59.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.245+0000 7f17a17fa640 10 monclient: dump: 2024-03-25T18:15:59.246 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:15:59.246 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:15:59.246 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:59.246 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:15:59.246 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:59.247 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:15:59.247 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:15:59.247 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:15:59.247 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:15:59.247 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:15:59.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.245+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f1790004c30 con 0x7f17a40b4d60 2024-03-25T18:15:59.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.246+0000 7f17a8b39640 10 monclient: get_auth_request con 0x7f178806dc00 auth_method 0 2024-03-25T18:15:59.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.246+0000 7f17a17fa640 10 client.4311.objecter ms_handle_connect 0x7f178806dc00 2024-03-25T18:15:59.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.252+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f17900546d0 con 0x7f17a40b4d60 2024-03-25T18:15:59.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.252+0000 7f17a17fa640 10 client.4311.objecter ms_dispatch 0x7f17a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:59.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.252+0000 7f17a17fa640 3 client.4311.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:59.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.252+0000 7f17a17fa640 3 client.4311.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:59.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.252+0000 7f17a17fa640 20 client.4311.objecter dump_active .. 0 homeless 2024-03-25T18:15:59.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.252+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17900501e0 con 0x7f17a40b4d60 2024-03-25T18:15:59.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.252+0000 7f17a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:59.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.252+0000 7f17a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:59.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.502+0000 7f17aad9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:59.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.502+0000 7f17aad9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:59.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.502+0000 7f17aad9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:15:59.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.502+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a40627b0 con 0x7f17a40b4d60 2024-03-25T18:15:59.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.531+0000 7f17a17fa640 1 -- 172.21.15.162:0/243164354 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f1790052110 con 0x7f17a40b4d60 2024-03-25T18:15:59.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.531+0000 7f17a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:59.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.531+0000 7f17a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:59.532 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:15:59.532 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:59.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 10 librados: watch_flush enter 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 10 librados: watch_flush exit 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 20 client.4311.objecter shutdown clearing up homeless session... 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 10 client.4311.objecter successfully canceled tick 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 >> v1:172.21.15.191:6800/1552703795 conn(0x7f178806dc00 legacy=0x7f17880700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 10 monclient: shutdown 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 >> v1:172.21.15.162:6789/0 conn(0x7f17a40b4d60 legacy=0x7f17a4131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 shutdown_connections 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 >> 172.21.15.162:0/243164354 conn(0x7f17a40617a0 msgr2=0x7f17a413c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 shutdown_connections 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 1 -- 172.21.15.162:0/243164354 wait complete. 2024-03-25T18:15:59.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:15:59.534+0000 7f17aad9d640 1 librados: shutdown 2024-03-25T18:15:59.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:02.211 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:14.438 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:02.219 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:14.447 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:02.253 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:14.481 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:02.261 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:14.489 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:02.281 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:14.508 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:02.282 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:14.498 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:02.287 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:14.504 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:02.306 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:14.534 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:02.313 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:14.529 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:06.353 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:06.466 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.465+0000 7fe85b84a640 10 monclient: get_monmap_and_config 2024-03-25T18:16:06.466 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.465+0000 7fe85b84a640 10 monclient: build_initial_monmap 2024-03-25T18:16:06.466 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.465+0000 7fe85b84a640 10 monclient: monmap: 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:16:06.466665+0000 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:16:06.466665+0000 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:16:06.467 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:06.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.466+0000 7fe85b84a640 1 Processor -- start 2024-03-25T18:16:06.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.466+0000 7fe85b84a640 1 -- start start 2024-03-25T18:16:06.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.466+0000 7fe85b84a640 10 monclient: init 2024-03-25T18:16:06.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:06.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:06.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 10 monclient(hunting): picked mon.noname-a con 0x7fe8540b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:06.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 10 monclient(hunting): picked mon.noname-b con 0x7fe8540b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:06.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 10 monclient(hunting): picked mon.noname-c con 0x7fe85414be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:06.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8540af940 con 0x7fe8540b4d60 2024-03-25T18:16:06.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8540b0a70 con 0x7fe85414be90 2024-03-25T18:16:06.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8540b0d50 con 0x7fe8540b1730 2024-03-25T18:16:06.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:06.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.467+0000 7fe85b84a640 10 monclient(hunting): authenticate will time out at 7665.741297s 2024-03-25T18:16:06.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.474+0000 7fe859de7640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe85414be90 0x7fe85414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:43734/0 (socket says 172.21.15.162:43734) 2024-03-25T18:16:06.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.474+0000 7fe859de7640 1 -- 172.21.15.162:0/2514180889 learned_addr learned my addr 172.21.15.162:0/2514180889 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:06.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.489+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172692824 0 0) 0x7fe8540b0a70 con 0x7fe85414be90 2024-03-25T18:16:06.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.489+0000 7fe84bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:06.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.489+0000 7fe84bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:06.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.489+0000 7fe84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:06.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.489+0000 7fe84bfff640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:16:06.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.489+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe840003520 con 0x7fe85414be90 2024-03-25T18:16:06.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.496+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076467093 0 0) 0x7fe8540b0d50 con 0x7fe8540b1730 2024-03-25T18:16:06.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.496+0000 7fe84bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:06.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.496+0000 7fe84bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:06.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.496+0000 7fe84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:06.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.496+0000 7fe84bfff640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:16:06.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.496+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8540b0a70 con 0x7fe8540b1730 2024-03-25T18:16:06.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.499+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733198845 0 0) 0x7fe8540af940 con 0x7fe8540b4d60 2024-03-25T18:16:06.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.499+0000 7fe84bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:06.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.499+0000 7fe84bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:06.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.499+0000 7fe84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:06.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.499+0000 7fe84bfff640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:16:06.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.499+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8540b0d50 con 0x7fe8540b4d60 2024-03-25T18:16:06.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537586813 0 0) 0x7fe840003520 con 0x7fe85414be90 2024-03-25T18:16:06.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 >> v1:172.21.15.162:6790/0 conn(0x7fe8540b1730 legacy=0x7fe8540b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:06.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 >> v1:172.21.15.162:6789/0 conn(0x7fe8540b4d60 legacy=0x7fe85413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:06.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:06.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 1 monclient: found mon.noname-c 2024-03-25T18:16:06.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:06.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8540b1030 con 0x7fe85414be90 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe84c001a10 con 0x7fe85414be90 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe84bfff640 10 monclient: dump: 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.503 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:06.504 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:06.504 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:06.504 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:06.504 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:06.504 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:06.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe85b84a640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:16:06.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.501+0000 7fe85b84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:06.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe84c001b70 con 0x7fe85414be90 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe84bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe84bfff640 1 -- 172.21.15.162:0/2514180889 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe84c001f80 con 0x7fe85414be90 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe85b84a640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe85b84a640 4 set_mon_vals no callback set 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe84bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe84bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe84bfff640 10 monclient: dump: 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.520 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.521 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:06.521 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:06.521 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:06.521 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:06.521 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:06.521 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe85b84a640 10 monclient: shutdown 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe85b84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe85b84a640 1 -- 172.21.15.162:0/2514180889 >> v1:172.21.15.191:6789/0 conn(0x7fe85414be90 legacy=0x7fe85414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.519+0000 7fe85b84a640 1 -- 172.21.15.162:0/2514180889 shutdown_connections 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.520+0000 7fe85b84a640 1 -- 172.21.15.162:0/2514180889 >> 172.21.15.162:0/2514180889 conn(0x7fe8540617a0 msgr2=0x7fe854061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.520+0000 7fe85b84a640 1 -- 172.21.15.162:0/2514180889 shutdown_connections 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.520+0000 7fe85b84a640 1 -- 172.21.15.162:0/2514180889 wait complete. 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 10 monclient: build_initial_monmap 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 10 monclient: monmap: 2024-03-25T18:16:06.523 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 1 librados: starting msgr at 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 1 librados: starting objecter 2024-03-25T18:16:06.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 1 Processor -- start 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 1 -- start start 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 1 librados: setting wanted keys 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 1 librados: calling monclient init 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.521+0000 7fe85b84a640 10 monclient: init 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 10 monclient(hunting): picked mon.noname-c con 0x7fe8540b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 10 monclient(hunting): picked mon.noname-b con 0x7fe8540b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 10 monclient(hunting): picked mon.noname-a con 0x7fe85414be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8541dad20 con 0x7fe85414be90 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe854138770 con 0x7fe8540b4d60 2024-03-25T18:16:06.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe854137450 con 0x7fe8540b1730 2024-03-25T18:16:06.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:06.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.522+0000 7fe85b84a640 10 monclient(hunting): authenticate will time out at 7665.796088s 2024-03-25T18:16:06.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.531+0000 7fe859de7640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe85414be90 0x7fe85412fd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:48814/0 (socket says 172.21.15.162:48814) 2024-03-25T18:16:06.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.531+0000 7fe859de7640 1 -- 172.21.15.162:0/382744791 learned_addr learned my addr 172.21.15.162:0/382744791 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:06.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.533+0000 7fe849ffb640 10 client.?.objecter ms_handle_connect 0x7fe85414be90 2024-03-25T18:16:06.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.533+0000 7fe849ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:06.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.534+0000 7fe849ffb640 10 client.?.objecter ms_handle_connect 0x7fe8540b1730 2024-03-25T18:16:06.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.534+0000 7fe849ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:06.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.534+0000 7fe849ffb640 10 client.?.objecter ms_handle_connect 0x7fe8540b4d60 2024-03-25T18:16:06.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.534+0000 7fe849ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:06.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.544+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114806884 0 0) 0x7fe8541dad20 con 0x7fe85414be90 2024-03-25T18:16:06.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.544+0000 7fe849ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:06.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.544+0000 7fe849ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:06.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.544+0000 7fe849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:06.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.544+0000 7fe849ffb640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:16:06.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.544+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe838004eb0 con 0x7fe85414be90 2024-03-25T18:16:06.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.549+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609440843 0 0) 0x7fe854138770 con 0x7fe8540b4d60 2024-03-25T18:16:06.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.549+0000 7fe849ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:06.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.549+0000 7fe849ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:06.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.549+0000 7fe849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:06.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.549+0000 7fe849ffb640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:16:06.551 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.549+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8541dad20 con 0x7fe8540b4d60 2024-03-25T18:16:06.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.554+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255391823 0 0) 0x7fe854137450 con 0x7fe8540b1730 2024-03-25T18:16:06.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.554+0000 7fe849ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:06.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.554+0000 7fe849ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:06.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.554+0000 7fe849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:06.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.554+0000 7fe849ffb640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:16:06.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.554+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe854138770 con 0x7fe8540b1730 2024-03-25T18:16:06.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.557+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1131728407 0 0) 0x7fe838004eb0 con 0x7fe85414be90 2024-03-25T18:16:06.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.557+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe854137450 con 0x7fe85414be90 2024-03-25T18:16:06.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.557+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe84c001ec0 con 0x7fe85414be90 2024-03-25T18:16:06.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.557+0000 7fe849ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:06.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.557+0000 7fe849ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:06.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.557+0000 7fe849ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:06.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.557+0000 7fe849ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.564+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 337585923 0 0) 0x7fe854138770 con 0x7fe8540b1730 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.564+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe838004eb0 con 0x7fe8540b1730 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.564+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe850001830 con 0x7fe8540b1730 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.564+0000 7fe849ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.564+0000 7fe849ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.564+0000 7fe849ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.565 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.566 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:06.566 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:06.566 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:06.566 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:06.566 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:06.566 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:06.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1036509970 0 0) 0x7fe838004eb0 con 0x7fe8540b1730 2024-03-25T18:16:06.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 >> v1:172.21.15.191:6789/0 conn(0x7fe8540b4d60 legacy=0x7fe8541db1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:06.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 >> v1:172.21.15.162:6789/0 conn(0x7fe85414be90 legacy=0x7fe85412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:06.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:06.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 1 monclient: found mon.c 2024-03-25T18:16:06.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:06.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8541375e0 con 0x7fe8540b1730 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe849ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe85b84a640 5 monclient: authenticate success, global_id 4307 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe85b84a640 10 monclient: _renew_subs 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe85b84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.569+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe854143900 con 0x7fe8540b1730 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.570+0000 7fe85b84a640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.570+0000 7fe85b84a640 10 monclient: _renew_subs 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.570+0000 7fe85b84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.570+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8541dbaa0 con 0x7fe8540b1730 2024-03-25T18:16:06.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.570+0000 7fe85b84a640 1 librados: init done 2024-03-25T18:16:06.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.571+0000 7fe85b84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:06.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.571+0000 7fe85b84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:06.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.571+0000 7fe85b84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:06.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.571+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8540627b0 con 0x7fe8540b1730 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.860+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe850001d40 con 0x7fe8540b1730 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.860+0000 7fe849ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.860+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe8500037b0 con 0x7fe8540b1730 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.860+0000 7fe849ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.860+0000 7fe849ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.860+0000 7fe849ffb640 10 monclient: dump: 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:06.861 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:06.862 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:06.862 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:06.862 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:06.862 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:06.862 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:06.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.860+0000 7fe84b7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:06.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.860+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fe8500197c0 con 0x7fe8540b1730 2024-03-25T18:16:06.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.861+0000 7fe858de5640 10 monclient: get_auth_request con 0x7fe83806dfb0 auth_method 0 2024-03-25T18:16:06.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.861+0000 7fe849ffb640 10 client.4307.objecter ms_handle_connect 0x7fe83806dfb0 2024-03-25T18:16:06.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.865+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fe850053510 con 0x7fe8540b1730 2024-03-25T18:16:06.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.865+0000 7fe849ffb640 10 client.4307.objecter ms_dispatch 0x7fe854000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:06.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.865+0000 7fe849ffb640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:06.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.865+0000 7fe849ffb640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:06.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.865+0000 7fe849ffb640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-25T18:16:06.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.865+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe85004ee50 con 0x7fe8540b1730 2024-03-25T18:16:06.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.865+0000 7fe849ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:06.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:06.865+0000 7fe849ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:07.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.079+0000 7fe85b84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:07.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.079+0000 7fe85b84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:07.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.079+0000 7fe85b84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:07.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.079+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe854069f40 con 0x7fe8540b1730 2024-03-25T18:16:07.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.144+0000 7fe849ffb640 1 -- 172.21.15.162:0/382744791 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fe850050f50 con 0x7fe8540b1730 2024-03-25T18:16:07.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.144+0000 7fe849ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:07.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.144+0000 7fe849ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:07.146 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:16:07.146 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 10 librados: watch_flush enter 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 10 librados: watch_flush exit 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 10 client.4307.objecter successfully canceled tick 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 >> v1:172.21.15.191:6800/1552703795 conn(0x7fe83806dfb0 legacy=0x7fe838070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 10 monclient: shutdown 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 >> v1:172.21.15.162:6790/0 conn(0x7fe8540b1730 legacy=0x7fe854136d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.147+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 shutdown_connections 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.148+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 >> 172.21.15.162:0/382744791 conn(0x7fe8540617a0 msgr2=0x7fe8540b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:07.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.148+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 shutdown_connections 2024-03-25T18:16:07.150 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.148+0000 7fe85b84a640 1 -- 172.21.15.162:0/382744791 wait complete. 2024-03-25T18:16:07.150 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:07.148+0000 7fe85b84a640 1 librados: shutdown 2024-03-25T18:16:07.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:13.967 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:14.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.088+0000 7f51b564e640 10 monclient: get_monmap_and_config 2024-03-25T18:16:14.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.088+0000 7f51b564e640 10 monclient: build_initial_monmap 2024-03-25T18:16:14.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.088+0000 7f51b564e640 10 monclient: monmap: 2024-03-25T18:16:14.089 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:14.089 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:14.090 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:16:14.089588+0000 2024-03-25T18:16:14.090 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:16:14.089588+0000 2024-03-25T18:16:14.090 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:14.090 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:14.090 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:14.090 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:16:14.090 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:16:14.090 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:14.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.089+0000 7f51b564e640 1 Processor -- start 2024-03-25T18:16:14.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.089+0000 7f51b564e640 1 -- start start 2024-03-25T18:16:14.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.089+0000 7f51b564e640 10 monclient: init 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 10 monclient(hunting): picked mon.noname-a con 0x7f51b00a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 10 monclient(hunting): picked mon.noname-b con 0x7f51b013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 10 monclient(hunting): picked mon.noname-c con 0x7f51b01430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b0142520 con 0x7f51b00a98f0 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b00b28a0 con 0x7f51b01430c0 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b00b2a30 con 0x7f51b013f710 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:14.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.090+0000 7f51b564e640 10 monclient(hunting): authenticate will time out at 7673.364058s 2024-03-25T18:16:14.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.105+0000 7f51af7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f51b01430c0 0x7f51b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:47546/0 (socket says 172.21.15.162:47546) 2024-03-25T18:16:14.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.105+0000 7f51af7fe640 1 -- 172.21.15.162:0/3434298914 learned_addr learned my addr 172.21.15.162:0/3434298914 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:14.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.118+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377470290 0 0) 0x7f51b00b28a0 con 0x7f51b01430c0 2024-03-25T18:16:14.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.118+0000 7f51adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.118+0000 7f51adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.118+0000 7f51adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.118+0000 7f51adffb640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:16:14.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.118+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5198003520 con 0x7f51b01430c0 2024-03-25T18:16:14.133 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.132+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205333198 0 0) 0x7f51b0142520 con 0x7f51b00a98f0 2024-03-25T18:16:14.133 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.132+0000 7f51adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.133 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.132+0000 7f51adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.133 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.132+0000 7f51adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.133 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.132+0000 7f51adffb640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:16:14.133 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.132+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b00b28a0 con 0x7f51b00a98f0 2024-03-25T18:16:14.136 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.135+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539914245 0 0) 0x7f51b00b2a30 con 0x7f51b013f710 2024-03-25T18:16:14.136 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.135+0000 7f51adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.136 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.135+0000 7f51adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.135+0000 7f51adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.135+0000 7f51adffb640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:16:14.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.135+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b0142520 con 0x7f51b013f710 2024-03-25T18:16:14.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3518807916 0 0) 0x7f5198003520 con 0x7f51b01430c0 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 >> v1:172.21.15.162:6790/0 conn(0x7f51b013f710 legacy=0x7f51b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 >> v1:172.21.15.162:6789/0 conn(0x7f51b00a98f0 legacy=0x7f51b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 1 monclient: found mon.noname-c 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b00b44c0 con 0x7f51b01430c0 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f51a8001a10 con 0x7f51b01430c0 2024-03-25T18:16:14.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51adffb640 10 monclient: dump: 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:14.142 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51b564e640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:16:14.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.139+0000 7f51b564e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:14.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.152+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51a8001b70 con 0x7f51b01430c0 2024-03-25T18:16:14.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.152+0000 7f51adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.152+0000 7f51adffb640 1 -- 172.21.15.162:0/3434298914 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f51a8001f80 con 0x7f51b01430c0 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.152+0000 7f51adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.152+0000 7f51adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.152+0000 7f51adffb640 10 monclient: dump: 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:14.154 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:14.155 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:14.155 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:14.155 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:14.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.152+0000 7f51b564e640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:14.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.152+0000 7f51b564e640 4 set_mon_vals no callback set 2024-03-25T18:16:14.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.153+0000 7f51b564e640 10 monclient: shutdown 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.153+0000 7f51b564e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.153+0000 7f51b564e640 1 -- 172.21.15.162:0/3434298914 >> v1:172.21.15.191:6789/0 conn(0x7f51b01430c0 legacy=0x7f51b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.153+0000 7f51b564e640 1 -- 172.21.15.162:0/3434298914 shutdown_connections 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.154+0000 7f51b564e640 1 -- 172.21.15.162:0/3434298914 >> 172.21.15.162:0/3434298914 conn(0x7f51b00b7ea0 msgr2=0x7f51b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.154+0000 7f51b564e640 1 -- 172.21.15.162:0/3434298914 shutdown_connections 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.154+0000 7f51b564e640 1 -- 172.21.15.162:0/3434298914 wait complete. 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.155+0000 7f51b564e640 10 monclient: build_initial_monmap 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.155+0000 7f51b564e640 10 monclient: monmap: 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:14.157 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.155+0000 7f51b564e640 1 librados: starting msgr at 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.155+0000 7f51b564e640 1 librados: starting objecter 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.155+0000 7f51b564e640 1 Processor -- start 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 1 -- start start 2024-03-25T18:16:14.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 1 librados: setting wanted keys 2024-03-25T18:16:14.159 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 1 librados: calling monclient init 2024-03-25T18:16:14.159 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 10 monclient: init 2024-03-25T18:16:14.159 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:14.159 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:14.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 10 monclient(hunting): picked mon.noname-a con 0x7f51b00a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:14.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 10 monclient(hunting): picked mon.noname-c con 0x7f51b013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:14.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 10 monclient(hunting): picked mon.noname-b con 0x7f51b01430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:14.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.156+0000 7f51b564e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b01369d0 con 0x7f51b00a98f0 2024-03-25T18:16:14.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.157+0000 7f51b564e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b0133a60 con 0x7f51b01430c0 2024-03-25T18:16:14.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.157+0000 7f51b564e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51b00ab9e0 con 0x7f51b013f710 2024-03-25T18:16:14.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.157+0000 7f51b564e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:14.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.157+0000 7f51b564e640 10 monclient(hunting): authenticate will time out at 7673.430675s 2024-03-25T18:16:14.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.169+0000 7f51af7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f51b01430c0 0x7f51b0133350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:47584/0 (socket says 172.21.15.162:47584) 2024-03-25T18:16:14.171 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.169+0000 7f51af7fe640 1 -- 172.21.15.162:0/2871910530 learned_addr learned my addr 172.21.15.162:0/2871910530 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:14.171 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.170+0000 7f51977fe640 10 client.?.objecter ms_handle_connect 0x7f51b01430c0 2024-03-25T18:16:14.171 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.170+0000 7f51977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:14.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.176+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202408519 0 0) 0x7f51b0133a60 con 0x7f51b01430c0 2024-03-25T18:16:14.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.176+0000 7f51977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.176+0000 7f51977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.176+0000 7f51977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.176+0000 7f51977fe640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:16:14.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.176+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519c004eb0 con 0x7f51b01430c0 2024-03-25T18:16:14.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.181+0000 7f51977fe640 10 client.?.objecter ms_handle_connect 0x7f51b013f710 2024-03-25T18:16:14.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.181+0000 7f51977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:14.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.182+0000 7f51977fe640 10 client.?.objecter ms_handle_connect 0x7f51b00a98f0 2024-03-25T18:16:14.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.182+0000 7f51977fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:14.186 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.185+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087284468 0 0) 0x7f51b00ab9e0 con 0x7f51b013f710 2024-03-25T18:16:14.186 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.185+0000 7f51977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.186 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.185+0000 7f51977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.186 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.185+0000 7f51977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.186 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.185+0000 7f51977fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:16:14.186 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.185+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b0133a60 con 0x7f51b013f710 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.196+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1532824191 0 0) 0x7f519c004eb0 con 0x7f51b01430c0 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.196+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51b00ab9e0 con 0x7f51b01430c0 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.196+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123767878 0 0) 0x7f51b01369d0 con 0x7f51b00a98f0 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.196+0000 7f51977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.196+0000 7f51977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.196+0000 7f51977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.196+0000 7f51977fe640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.196+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f519c004eb0 con 0x7f51b00a98f0 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.197+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f51a8001f60 con 0x7f51b01430c0 2024-03-25T18:16:14.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.197+0000 7f51977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.197+0000 7f51977fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.197+0000 7f51977fe640 10 monclient(hunting): dump: 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:14.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.197+0000 7f51977fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:14.201 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.200+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015539835 0 0) 0x7f51b0133a60 con 0x7f51b013f710 2024-03-25T18:16:14.201 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.200+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51b01369d0 con 0x7f51b013f710 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.200+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f51a4001870 con 0x7f51b013f710 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.200+0000 7f51977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.200+0000 7f51977fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.200+0000 7f51977fe640 10 monclient(hunting): dump: 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:14.202 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:14.203 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2936175774 0 0) 0x7f51b00ab9e0 con 0x7f51b01430c0 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 >> v1:172.21.15.162:6790/0 conn(0x7f51b013f710 legacy=0x7f51b01362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 >> v1:172.21.15.162:6789/0 conn(0x7f51b00a98f0 legacy=0x7f51b0132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 1 monclient: found mon.b 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b00abcc0 con 0x7f51b01430c0 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:14.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:16:14.205 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 10 monclient: _renew_subs 2024-03-25T18:16:14.205 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:14.205 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51b00abfa0 con 0x7f51b01430c0 2024-03-25T18:16:14.205 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:14.205 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 10 monclient: _renew_subs 2024-03-25T18:16:14.205 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:14.205 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51b0137e40 con 0x7f51b01430c0 2024-03-25T18:16:14.205 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.202+0000 7f51b564e640 1 librados: init done 2024-03-25T18:16:14.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.204+0000 7f51b564e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:14.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.204+0000 7f51b564e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:14.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.204+0000 7f51b564e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:14.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.204+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51b0139060 con 0x7f51b01430c0 2024-03-25T18:16:14.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.488+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51a80020c0 con 0x7f51b01430c0 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.488+0000 7f51977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.488+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f51a8004fb0 con 0x7f51b01430c0 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.488+0000 7f51977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.488+0000 7f51977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.488+0000 7f51977fe640 10 monclient: dump: 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:14.490 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:14.491 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:14.491 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:14.491 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:14.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.488+0000 7f51ad7fa640 4 set_mon_vals no callback set 2024-03-25T18:16:14.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.489+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f51a801afc0 con 0x7f51b01430c0 2024-03-25T18:16:14.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.490+0000 7f51aeffd640 10 monclient: get_auth_request con 0x7f519c06df30 auth_method 0 2024-03-25T18:16:14.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.490+0000 7f51977fe640 10 client.4309.objecter ms_handle_connect 0x7f519c06df30 2024-03-25T18:16:14.496 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.495+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f51a8054c50 con 0x7f51b01430c0 2024-03-25T18:16:14.496 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.495+0000 7f51977fe640 10 client.4309.objecter ms_dispatch 0x7f51b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:14.496 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.495+0000 7f51977fe640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:14.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.495+0000 7f51977fe640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:14.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.495+0000 7f51977fe640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T18:16:14.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.495+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51a8050590 con 0x7f51b01430c0 2024-03-25T18:16:14.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.495+0000 7f51977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:14.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.495+0000 7f51977fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:14.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.730+0000 7f51b564e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:14.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.730+0000 7f51b564e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:14.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.730+0000 7f51b564e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:14.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.730+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51b013dfd0 con 0x7f51b01430c0 2024-03-25T18:16:14.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.758+0000 7f51977fe640 1 -- 172.21.15.162:0/2871910530 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f51a8052690 con 0x7f51b01430c0 2024-03-25T18:16:14.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.758+0000 7f51977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:14.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.758+0000 7f51977fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:14.759 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:16:14.760 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:14.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.760+0000 7f51b564e640 10 librados: watch_flush enter 2024-03-25T18:16:14.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.760+0000 7f51b564e640 10 librados: watch_flush exit 2024-03-25T18:16:14.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.760+0000 7f51b564e640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T18:16:14.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.760+0000 7f51b564e640 10 client.4309.objecter successfully canceled tick 2024-03-25T18:16:14.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 >> v1:172.21.15.191:6800/1552703795 conn(0x7f519c06df30 legacy=0x7f519c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 10 monclient: shutdown 2024-03-25T18:16:14.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:14.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 >> v1:172.21.15.191:6789/0 conn(0x7f51b01430c0 legacy=0x7f51b0133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 shutdown_connections 2024-03-25T18:16:14.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 >> 172.21.15.162:0/2871910530 conn(0x7f51b00b7ea0 msgr2=0x7f51b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:14.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 shutdown_connections 2024-03-25T18:16:14.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 1 -- 172.21.15.162:0/2871910530 wait complete. 2024-03-25T18:16:14.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:14.761+0000 7f51b564e640 1 librados: shutdown 2024-03-25T18:16:14.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:21.580 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:21.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.696+0000 7f91e4d13640 10 monclient: get_monmap_and_config 2024-03-25T18:16:21.697 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.696+0000 7f91e4d13640 10 monclient: build_initial_monmap 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.696+0000 7f91e4d13640 10 monclient: monmap: 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:16:21.697687+0000 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:16:21.697687+0000 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:16:21.698 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.697+0000 7f91e4d13640 1 Processor -- start 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.697+0000 7f91e4d13640 1 -- start start 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.697+0000 7f91e4d13640 10 monclient: init 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 10 monclient(hunting): picked mon.noname-b con 0x7f91e00a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 10 monclient(hunting): picked mon.noname-c con 0x7f91e013f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 10 monclient(hunting): picked mon.noname-a con 0x7f91e01430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e0142520 con 0x7f91e01430c0 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00b28a0 con 0x7f91e013f710 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00b2a30 con 0x7f91e00a98f0 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:21.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.698+0000 7f91e4d13640 10 monclient(hunting): authenticate will time out at 7680.971988s 2024-03-25T18:16:21.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.708+0000 7f91ddd9b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f91e013f710 0x7f91e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:52280/0 (socket says 172.21.15.162:52280) 2024-03-25T18:16:21.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.708+0000 7f91ddd9b640 1 -- 172.21.15.162:0/4153929325 learned_addr learned my addr 172.21.15.162:0/4153929325 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:21.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952041255 0 0) 0x7f91e0142520 con 0x7f91e01430c0 2024-03-25T18:16:21.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:21.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:21.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:21.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:16:21.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c8003520 con 0x7f91e01430c0 2024-03-25T18:16:21.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 784601277 0 0) 0x7f91e00b28a0 con 0x7f91e013f710 2024-03-25T18:16:21.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:21.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:21.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:21.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:16:21.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.725+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e0142520 con 0x7f91e013f710 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.735+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 891337101 0 0) 0x7f91c8003520 con 0x7f91e01430c0 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 >> v1:172.21.15.162:6790/0 conn(0x7f91e00a98f0 legacy=0x7f91e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 >> v1:172.21.15.191:6789/0 conn(0x7f91e013f710 legacy=0x7f91e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 1 monclient: found mon.noname-a 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00b44c0 con 0x7f91e01430c0 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 10 monclient: _finish_auth 0 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f91d8001a10 con 0x7f91e01430c0 2024-03-25T18:16:21.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91dd59a640 10 monclient: dump: 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:21.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91e4d13640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:16:21.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.736+0000 7f91e4d13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:21.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.748+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91d8001b70 con 0x7f91e01430c0 2024-03-25T18:16:21.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.748+0000 7f91dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:21.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.748+0000 7f91dd59a640 1 -- 172.21.15.162:0/4153929325 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f91d8001f80 con 0x7f91e01430c0 2024-03-25T18:16:21.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.748+0000 7f91e4d13640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:21.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.748+0000 7f91e4d13640 4 set_mon_vals no callback set 2024-03-25T18:16:21.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.748+0000 7f91dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.748+0000 7f91dd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.748+0000 7f91dd59a640 10 monclient: dump: 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:21.750 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.749+0000 7f91e4d13640 10 monclient: shutdown 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.749+0000 7f91e4d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.749+0000 7f91e4d13640 1 -- 172.21.15.162:0/4153929325 >> v1:172.21.15.162:6789/0 conn(0x7f91e01430c0 legacy=0x7f91e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.749+0000 7f91e4d13640 1 -- 172.21.15.162:0/4153929325 shutdown_connections 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.749+0000 7f91e4d13640 1 -- 172.21.15.162:0/4153929325 >> 172.21.15.162:0/4153929325 conn(0x7f91e00b7ea0 msgr2=0x7f91e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.749+0000 7f91e4d13640 1 -- 172.21.15.162:0/4153929325 shutdown_connections 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.749+0000 7f91e4d13640 1 -- 172.21.15.162:0/4153929325 wait complete. 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.750+0000 7f91e4d13640 10 monclient: build_initial_monmap 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.750+0000 7f91e4d13640 10 monclient: monmap: 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:21.753 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.750+0000 7f91e4d13640 1 librados: starting msgr at 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.750+0000 7f91e4d13640 1 librados: starting objecter 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.750+0000 7f91e4d13640 1 Processor -- start 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 1 -- start start 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 1 librados: setting wanted keys 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 1 librados: calling monclient init 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 10 monclient: init 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:21.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:21.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 10 monclient(hunting): picked mon.noname-c con 0x7f91e00a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:21.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 10 monclient(hunting): picked mon.noname-b con 0x7f91e013f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:21.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 10 monclient(hunting): picked mon.noname-a con 0x7f91e01430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:21.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e0136c50 con 0x7f91e01430c0 2024-03-25T18:16:21.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e0136ec0 con 0x7f91e013f710 2024-03-25T18:16:21.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e01317f0 con 0x7f91e00a98f0 2024-03-25T18:16:21.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:21.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.751+0000 7f91e4d13640 10 monclient(hunting): authenticate will time out at 7681.025412s 2024-03-25T18:16:21.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.765+0000 7f91ded9d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f91e01430c0 0x7f91e01db380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47122/0 (socket says 172.21.15.162:47122) 2024-03-25T18:16:21.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.766+0000 7f91ded9d640 1 -- 172.21.15.162:0/3704851515 learned_addr learned my addr 172.21.15.162:0/3704851515 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:21.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.767+0000 7f91ceffd640 10 client.?.objecter ms_handle_connect 0x7f91e00a98f0 2024-03-25T18:16:21.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.767+0000 7f91ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:21.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.767+0000 7f91ceffd640 10 client.?.objecter ms_handle_connect 0x7f91e01430c0 2024-03-25T18:16:21.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.767+0000 7f91ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:21.773 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.772+0000 7f91ceffd640 10 client.?.objecter ms_handle_connect 0x7f91e013f710 2024-03-25T18:16:21.773 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.772+0000 7f91ceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:21.773 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.773+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198241019 0 0) 0x7f91e01317f0 con 0x7f91e00a98f0 2024-03-25T18:16:21.774 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.773+0000 7f91ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:21.774 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.773+0000 7f91ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:21.774 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.773+0000 7f91ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:21.774 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.773+0000 7f91ceffd640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:16:21.774 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.773+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0004eb0 con 0x7f91e00a98f0 2024-03-25T18:16:21.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.787+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334601893 0 0) 0x7f91e0136c50 con 0x7f91e01430c0 2024-03-25T18:16:21.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.787+0000 7f91ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:21.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.787+0000 7f91ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:21.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.787+0000 7f91ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:21.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.787+0000 7f91ceffd640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:16:21.788 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.787+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e01317f0 con 0x7f91e01430c0 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.791+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 868525395 0 0) 0x7f91c0004eb0 con 0x7f91e00a98f0 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.791+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91e0136c50 con 0x7f91e00a98f0 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.791+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f91d00018c0 con 0x7f91e00a98f0 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.791+0000 7f91ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.791+0000 7f91ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.791+0000 7f91ceffd640 10 monclient(hunting): dump: 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:21.792 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:21.793 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:21.793 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:21.793 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:21.793 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:21.793 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:21.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.791+0000 7f91ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:21.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.795+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352812595 0 0) 0x7f91e0136ec0 con 0x7f91e013f710 2024-03-25T18:16:21.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.795+0000 7f91ceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:21.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.795+0000 7f91ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:21.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.795+0000 7f91ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:21.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.795+0000 7f91ceffd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:16:21.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.795+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c0004eb0 con 0x7f91e013f710 2024-03-25T18:16:21.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 132135437 0 0) 0x7f91e0136c50 con 0x7f91e00a98f0 2024-03-25T18:16:21.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 >> v1:172.21.15.191:6789/0 conn(0x7f91e013f710 legacy=0x7f91e01dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 >> v1:172.21.15.162:6789/0 conn(0x7f91e01430c0 legacy=0x7f91e01db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 1 monclient: found mon.c 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0131980 con 0x7f91e00a98f0 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 5 monclient: authenticate success, global_id 4319 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 10 monclient: _renew_subs 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e0135070 con 0x7f91e00a98f0 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:21.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 10 monclient: _renew_subs 2024-03-25T18:16:21.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:21.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e00b7870 con 0x7f91e00a98f0 2024-03-25T18:16:21.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.799+0000 7f91e4d13640 1 librados: init done 2024-03-25T18:16:21.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.800+0000 7f91e4d13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:21.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.800+0000 7f91e4d13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:21.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.800+0000 7f91e4d13640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:21.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:21.800+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e0135c70 con 0x7f91e00a98f0 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.095+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91d00035f0 con 0x7f91e00a98f0 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.096+0000 7f91ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.096+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f91d0003980 con 0x7f91e00a98f0 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.096+0000 7f91ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.096+0000 7f91ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.096+0000 7f91ceffd640 10 monclient: dump: 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:22.097 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:22.098 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:22.098 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:22.098 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:22.098 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:22.098 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:22.098 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:22.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.096+0000 7f91dcd99640 4 set_mon_vals no callback set 2024-03-25T18:16:22.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.096+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f91d0003ba0 con 0x7f91e00a98f0 2024-03-25T18:16:22.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.097+0000 7f91ddd9b640 10 monclient: get_auth_request con 0x7f91c006de00 auth_method 0 2024-03-25T18:16:22.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.097+0000 7f91ceffd640 10 client.4319.objecter ms_handle_connect 0x7f91c006de00 2024-03-25T18:16:22.105 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.103+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f91d0053570 con 0x7f91e00a98f0 2024-03-25T18:16:22.105 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.103+0000 7f91ceffd640 10 client.4319.objecter ms_dispatch 0x7f91e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:22.105 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.103+0000 7f91ceffd640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:22.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.103+0000 7f91ceffd640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:22.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.103+0000 7f91ceffd640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-25T18:16:22.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.103+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91d004f080 con 0x7f91e00a98f0 2024-03-25T18:16:22.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.103+0000 7f91ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:22.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.103+0000 7f91ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:22.278 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:34.506 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:22.295 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:34.523 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:22.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.334+0000 7f91e4d13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:22.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.334+0000 7f91e4d13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:22.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.334+0000 7f91e4d13640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:22.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.334+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91e00627b0 con 0x7f91e00a98f0 2024-03-25T18:16:22.353 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:34.580 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:22.365 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:34.582 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:22.368 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:34.596 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:22.374 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:34.591 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:22.388 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:34.616 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:22.394 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.393+0000 7f91ceffd640 1 -- 172.21.15.162:0/3704851515 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f91d0050fb0 con 0x7f91e00a98f0 2024-03-25T18:16:22.394 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.393+0000 7f91ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:22.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.393+0000 7f91ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:22.395 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:16:22.395 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 10 librados: watch_flush enter 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 10 librados: watch_flush exit 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 10 client.4319.objecter successfully canceled tick 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 >> v1:172.21.15.191:6800/1552703795 conn(0x7f91c006de00 legacy=0x7f91c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 10 monclient: shutdown 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 >> v1:172.21.15.162:6790/0 conn(0x7f91e00a98f0 legacy=0x7f91e01310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 shutdown_connections 2024-03-25T18:16:22.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 >> 172.21.15.162:0/3704851515 conn(0x7f91e00b7ea0 msgr2=0x7f91e013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:22.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 shutdown_connections 2024-03-25T18:16:22.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 1 -- 172.21.15.162:0/3704851515 wait complete. 2024-03-25T18:16:22.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:22.396+0000 7f91e4d13640 1 librados: shutdown 2024-03-25T18:16:22.399 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:34.615 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:22.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:22.423 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:34.651 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:29.215 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:29.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.329+0000 7f0d8a70e640 10 monclient: get_monmap_and_config 2024-03-25T18:16:29.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.329+0000 7f0d8a70e640 10 monclient: build_initial_monmap 2024-03-25T18:16:29.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.329+0000 7f0d8a70e640 10 monclient: monmap: 2024-03-25T18:16:29.330 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:29.330 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:29.330 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:16:29.330702+0000 2024-03-25T18:16:29.331 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:16:29.330702+0000 2024-03-25T18:16:29.331 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:29.331 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:29.331 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:29.331 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:16:29.331 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:16:29.331 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.330+0000 7f0d8a70e640 1 Processor -- start 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.330+0000 7f0d8a70e640 1 -- start start 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.330+0000 7f0d8a70e640 10 monclient: init 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 10 monclient(hunting): picked mon.noname-a con 0x7f0d840b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 10 monclient(hunting): picked mon.noname-c con 0x7f0d840b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 10 monclient(hunting): picked mon.noname-b con 0x7f0d8414be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840af940 con 0x7f0d840b4d60 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840b0a70 con 0x7f0d840b1730 2024-03-25T18:16:29.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840b0d50 con 0x7f0d8414be90 2024-03-25T18:16:29.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:29.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.331+0000 7f0d8a70e640 10 monclient(hunting): authenticate will time out at 7688.604845s 2024-03-25T18:16:29.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.336+0000 7f0d837fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f0d840b1730 0x7f0d840b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:35798/0 (socket says 172.21.15.162:35798) 2024-03-25T18:16:29.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.336+0000 7f0d837fe640 1 -- 172.21.15.162:0/1957229273 learned_addr learned my addr 172.21.15.162:0/1957229273 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:29.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.347+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588486462 0 0) 0x7f0d840b0a70 con 0x7f0d840b1730 2024-03-25T18:16:29.348 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.347+0000 7f0d82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.348 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.347+0000 7f0d82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.348 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.347+0000 7f0d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.348 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.347+0000 7f0d82ffd640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:16:29.348 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.347+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c003520 con 0x7f0d840b1730 2024-03-25T18:16:29.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.358+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246107855 0 0) 0x7f0d840b0d50 con 0x7f0d8414be90 2024-03-25T18:16:29.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.358+0000 7f0d82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.358+0000 7f0d82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.358+0000 7f0d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.358+0000 7f0d82ffd640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:16:29.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.358+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d840b0a70 con 0x7f0d8414be90 2024-03-25T18:16:29.365 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.364+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115005546 0 0) 0x7f0d840af940 con 0x7f0d840b4d60 2024-03-25T18:16:29.365 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.364+0000 7f0d82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.365 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.364+0000 7f0d82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.365 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.364+0000 7f0d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.365 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.364+0000 7f0d82ffd640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:16:29.365 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.364+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d840b0d50 con 0x7f0d840b4d60 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4078160848 0 0) 0x7f0d6c003520 con 0x7f0d840b1730 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 >> v1:172.21.15.162:6790/0 conn(0x7f0d8414be90 legacy=0x7f0d8414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 >> v1:172.21.15.162:6789/0 conn(0x7f0d840b4d60 legacy=0x7f0d8413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 1 monclient: found mon.noname-c 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d840b1030 con 0x7f0d840b1730 2024-03-25T18:16:29.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0d78001690 con 0x7f0d840b1730 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d8a70e640 5 monclient: authenticate success, global_id 4318 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d8a70e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d82ffd640 10 monclient: dump: 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:29.378 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:29.379 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:29.379 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:29.379 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:29.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.376+0000 7f0d8a70e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:29.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.411+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d780017f0 con 0x7f0d840b1730 2024-03-25T18:16:29.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.411+0000 7f0d82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.411+0000 7f0d82ffd640 1 -- 172.21.15.162:0/1957229273 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0d78001c00 con 0x7f0d840b1730 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.411+0000 7f0d8a70e640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.411+0000 7f0d8a70e640 4 set_mon_vals no callback set 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.411+0000 7f0d82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.411+0000 7f0d82ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.411+0000 7f0d82ffd640 10 monclient: dump: 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:29.413 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:29.415 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.412+0000 7f0d8a70e640 10 monclient: shutdown 2024-03-25T18:16:29.415 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.412+0000 7f0d8a70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:29.415 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.412+0000 7f0d8a70e640 1 -- 172.21.15.162:0/1957229273 >> v1:172.21.15.191:6789/0 conn(0x7f0d840b1730 legacy=0x7f0d840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.412+0000 7f0d8a70e640 1 -- 172.21.15.162:0/1957229273 shutdown_connections 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.412+0000 7f0d8a70e640 1 -- 172.21.15.162:0/1957229273 >> 172.21.15.162:0/1957229273 conn(0x7f0d840617a0 msgr2=0x7f0d84061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.412+0000 7f0d8a70e640 1 -- 172.21.15.162:0/1957229273 shutdown_connections 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.412+0000 7f0d8a70e640 1 -- 172.21.15.162:0/1957229273 wait complete. 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 10 monclient: build_initial_monmap 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 10 monclient: monmap: 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:16:29.416 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 1 librados: starting msgr at 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 1 librados: starting objecter 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 1 Processor -- start 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 1 -- start start 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 1 librados: setting wanted keys 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 1 librados: calling monclient init 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.413+0000 7f0d8a70e640 10 monclient: init 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 10 monclient(hunting): picked mon.noname-b con 0x7f0d840b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 10 monclient(hunting): picked mon.noname-a con 0x7f0d840b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 10 monclient(hunting): picked mon.noname-c con 0x7f0d8414be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d84141790 con 0x7f0d840b4d60 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d84141a00 con 0x7f0d840b1730 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d841d7a90 con 0x7f0d8414be90 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:29.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.414+0000 7f0d8a70e640 10 monclient(hunting): authenticate will time out at 7688.688204s 2024-03-25T18:16:29.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.423+0000 7f0d83fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f0d840b1730 0x7f0d8413b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:35884/0 (socket says 172.21.15.162:35884) 2024-03-25T18:16:29.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.423+0000 7f0d83fff640 1 -- 172.21.15.162:0/312412473 learned_addr learned my addr 172.21.15.162:0/312412473 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:29.428 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.428+0000 7f0d80ff9640 10 client.?.objecter ms_handle_connect 0x7f0d840b1730 2024-03-25T18:16:29.428 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.428+0000 7f0d80ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:29.431 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.430+0000 7f0d80ff9640 10 client.?.objecter ms_handle_connect 0x7f0d840b4d60 2024-03-25T18:16:29.431 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.430+0000 7f0d80ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:29.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.431+0000 7f0d80ff9640 10 client.?.objecter ms_handle_connect 0x7f0d8414be90 2024-03-25T18:16:29.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.431+0000 7f0d80ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:29.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.434+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697524199 0 0) 0x7f0d84141a00 con 0x7f0d840b1730 2024-03-25T18:16:29.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.434+0000 7f0d80ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.434+0000 7f0d80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.434+0000 7f0d80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.434+0000 7f0d80ff9640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:16:29.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.434+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d70004eb0 con 0x7f0d840b1730 2024-03-25T18:16:29.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.439+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15056807 0 0) 0x7f0d84141790 con 0x7f0d840b4d60 2024-03-25T18:16:29.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.439+0000 7f0d80ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.439+0000 7f0d80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.439+0000 7f0d80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.439+0000 7f0d80ff9640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:16:29.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.439+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d84141a00 con 0x7f0d840b4d60 2024-03-25T18:16:29.442 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.441+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329503645 0 0) 0x7f0d841d7a90 con 0x7f0d8414be90 2024-03-25T18:16:29.442 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.441+0000 7f0d80ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.442 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.441+0000 7f0d80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.442 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.441+0000 7f0d80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.442 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.441+0000 7f0d80ff9640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:16:29.443 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.441+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d84141790 con 0x7f0d8414be90 2024-03-25T18:16:29.443 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.442+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 838849940 0 0) 0x7f0d70004eb0 con 0x7f0d840b1730 2024-03-25T18:16:29.443 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.442+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d841d7a90 con 0x7f0d840b1730 2024-03-25T18:16:29.443 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.442+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0d74001a10 con 0x7f0d840b1730 2024-03-25T18:16:29.443 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.442+0000 7f0d80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.443 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.442+0000 7f0d80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:29.443 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.442+0000 7f0d80ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:29.443 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:29.444 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.442+0000 7f0d80ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:29.452 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 573042769 0 0) 0x7f0d841d7a90 con 0x7f0d840b1730 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 >> v1:172.21.15.162:6790/0 conn(0x7f0d8414be90 legacy=0x7f0d841d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 >> v1:172.21.15.162:6789/0 conn(0x7f0d840b4d60 legacy=0x7f0d84142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 1 monclient: found mon.b 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84143140 con 0x7f0d840b1730 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 10 monclient: _renew_subs 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d841d86c0 con 0x7f0d840b1730 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:29.453 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 10 monclient: _renew_subs 2024-03-25T18:16:29.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:29.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d841422e0 con 0x7f0d840b1730 2024-03-25T18:16:29.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.451+0000 7f0d8a70e640 1 librados: init done 2024-03-25T18:16:29.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.452+0000 7f0d8a70e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:29.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.452+0000 7f0d8a70e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:29.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.452+0000 7f0d8a70e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:29.455 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.452+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d841d7c20 con 0x7f0d840b1730 2024-03-25T18:16:29.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.726+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d74001c70 con 0x7f0d840b1730 2024-03-25T18:16:29.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.726+0000 7f0d80ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:29.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.726+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0d74003a60 con 0x7f0d840b1730 2024-03-25T18:16:29.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.726+0000 7f0d80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.726+0000 7f0d80ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.726+0000 7f0d80ff9640 10 monclient: dump: 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:29.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.726+0000 7f0d827fc640 4 set_mon_vals no callback set 2024-03-25T18:16:29.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.727+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f0d74003c80 con 0x7f0d840b1730 2024-03-25T18:16:29.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.728+0000 7f0d837fe640 10 monclient: get_auth_request con 0x7f0d7006ddf0 auth_method 0 2024-03-25T18:16:29.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.728+0000 7f0d80ff9640 10 client.4321.objecter ms_handle_connect 0x7f0d7006ddf0 2024-03-25T18:16:29.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.732+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f0d74054860 con 0x7f0d840b1730 2024-03-25T18:16:29.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.732+0000 7f0d80ff9640 10 client.4321.objecter ms_dispatch 0x7f0d84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:29.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.732+0000 7f0d80ff9640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:29.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.732+0000 7f0d80ff9640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:29.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.732+0000 7f0d80ff9640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T18:16:29.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.733+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d74050370 con 0x7f0d840b1730 2024-03-25T18:16:29.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.733+0000 7f0d80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:29.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.733+0000 7f0d80ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:29.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.965+0000 7f0d8a70e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:29.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.965+0000 7f0d8a70e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:29.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.965+0000 7f0d8a70e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:29.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.965+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d840627b0 con 0x7f0d840b1730 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.986+0000 7f0d80ff9640 1 -- 172.21.15.162:0/312412473 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f0d740522a0 con 0x7f0d840b1730 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.986+0000 7f0d80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.986+0000 7f0d80ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:29.987 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:16:29.988 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:29.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 10 librados: watch_flush enter 2024-03-25T18:16:29.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 10 librados: watch_flush exit 2024-03-25T18:16:29.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T18:16:29.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 10 client.4321.objecter successfully canceled tick 2024-03-25T18:16:29.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 >> v1:172.21.15.191:6800/1552703795 conn(0x7f0d7006ddf0 legacy=0x7f0d700702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 10 monclient: shutdown 2024-03-25T18:16:29.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:29.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 >> v1:172.21.15.191:6789/0 conn(0x7f0d840b1730 legacy=0x7f0d8413b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 shutdown_connections 2024-03-25T18:16:29.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 >> 172.21.15.162:0/312412473 conn(0x7f0d840617a0 msgr2=0x7f0d840b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:29.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 shutdown_connections 2024-03-25T18:16:29.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 1 -- 172.21.15.162:0/312412473 wait complete. 2024-03-25T18:16:29.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:29.989+0000 7f0d8a70e640 1 librados: shutdown 2024-03-25T18:16:30.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:36.809 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.923+0000 7f147a28f640 10 monclient: get_monmap_and_config 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.923+0000 7f147a28f640 10 monclient: build_initial_monmap 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.923+0000 7f147a28f640 10 monclient: monmap: 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:16:36.924335+0000 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:16:36.924335+0000 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:16:36.924 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 1 Processor -- start 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 1 -- start start 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 10 monclient: init 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 10 monclient(hunting): picked mon.noname-b con 0x7f14740b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 10 monclient(hunting): picked mon.noname-a con 0x7f14740b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 10 monclient(hunting): picked mon.noname-c con 0x7f147414be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14740af940 con 0x7f14740b1730 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14740b0a70 con 0x7f147414be90 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14740b0d50 con 0x7f14740b4d60 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:36.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.924+0000 7f147a28f640 10 monclient(hunting): authenticate will time out at 7696.198480s 2024-03-25T18:16:36.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.935+0000 7f146b7fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f14740b1730 0x7f14740b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:60676/0 (socket says 172.21.15.162:60676) 2024-03-25T18:16:36.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.935+0000 7f146b7fe640 1 -- 172.21.15.162:0/1412475523 learned_addr learned my addr 172.21.15.162:0/1412475523 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:36.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476065977 0 0) 0x7f14740af940 con 0x7f14740b1730 2024-03-25T18:16:36.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:36.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:36.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:36.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:16:36.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1460003520 con 0x7f14740b1730 2024-03-25T18:16:36.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001338956 0 0) 0x7f14740b0a70 con 0x7f147414be90 2024-03-25T18:16:36.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:36.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:36.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:36.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:16:36.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.950+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14740af940 con 0x7f147414be90 2024-03-25T18:16:36.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.968+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707960464 0 0) 0x7f14740b0d50 con 0x7f14740b4d60 2024-03-25T18:16:36.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.968+0000 7f14737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:36.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.968+0000 7f14737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:36.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.968+0000 7f14737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:36.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.968+0000 7f14737fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:16:36.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.968+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14740b0a70 con 0x7f14740b4d60 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893264207 0 0) 0x7f14740af940 con 0x7f147414be90 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 >> v1:172.21.15.162:6790/0 conn(0x7f14740b4d60 legacy=0x7f147413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 >> v1:172.21.15.162:6789/0 conn(0x7f14740b1730 legacy=0x7f14740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 1 monclient: found mon.noname-c 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14740b1030 con 0x7f147414be90 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:36.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f146c0019f0 con 0x7f147414be90 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f147a28f640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f147a28f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f14737fe640 10 monclient: dump: 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:36.974 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:36.975 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:36.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.971+0000 7f147a28f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:36.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.995+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f146c001b50 con 0x7f147414be90 2024-03-25T18:16:36.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.995+0000 7f14737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:36.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.995+0000 7f14737fe640 1 -- 172.21.15.162:0/1412475523 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f146c001f60 con 0x7f147414be90 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.995+0000 7f14737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.995+0000 7f14737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.995+0000 7f14737fe640 10 monclient: dump: 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.995+0000 7f147a28f640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:36.997 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.995+0000 7f147a28f640 4 set_mon_vals no callback set 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.996+0000 7f147a28f640 10 monclient: shutdown 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.996+0000 7f147a28f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.996+0000 7f147a28f640 1 -- 172.21.15.162:0/1412475523 >> v1:172.21.15.191:6789/0 conn(0x7f147414be90 legacy=0x7f147414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.996+0000 7f147a28f640 1 -- 172.21.15.162:0/1412475523 shutdown_connections 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.996+0000 7f147a28f640 1 -- 172.21.15.162:0/1412475523 >> 172.21.15.162:0/1412475523 conn(0x7f14740617a0 msgr2=0x7f1474061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.997+0000 7f147a28f640 1 -- 172.21.15.162:0/1412475523 shutdown_connections 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.997+0000 7f147a28f640 1 -- 172.21.15.162:0/1412475523 wait complete. 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.997+0000 7f147a28f640 10 monclient: build_initial_monmap 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.997+0000 7f147a28f640 10 monclient: monmap: 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:36.999 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.998+0000 7f147a28f640 1 librados: starting msgr at 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.998+0000 7f147a28f640 1 librados: starting objecter 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.998+0000 7f147a28f640 1 Processor -- start 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.998+0000 7f147a28f640 1 -- start start 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.998+0000 7f147a28f640 1 librados: setting wanted keys 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.998+0000 7f147a28f640 1 librados: calling monclient init 2024-03-25T18:16:37.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.998+0000 7f147a28f640 10 monclient: init 2024-03-25T18:16:37.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:37.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:37.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 10 monclient(hunting): picked mon.noname-a con 0x7f14740b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:37.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 10 monclient(hunting): picked mon.noname-c con 0x7f14740b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:37.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 10 monclient(hunting): picked mon.noname-b con 0x7f147414be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:37.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14741406f0 con 0x7f14740b1730 2024-03-25T18:16:37.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1474140380 con 0x7f147414be90 2024-03-25T18:16:37.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1474144ce0 con 0x7f14740b4d60 2024-03-25T18:16:37.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:37.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:36.999+0000 7f147a28f640 10 monclient(hunting): authenticate will time out at 7696.273378s 2024-03-25T18:16:37.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.006+0000 7f147882c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f147414be90 0x7f14741d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:59308/0 (socket says 172.21.15.162:59308) 2024-03-25T18:16:37.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.006+0000 7f147882c640 1 -- 172.21.15.162:0/772612679 learned_addr learned my addr 172.21.15.162:0/772612679 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:37.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.008+0000 7f14717fa640 10 client.?.objecter ms_handle_connect 0x7f147414be90 2024-03-25T18:16:37.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.008+0000 7f14717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.015+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866683273 0 0) 0x7f1474140380 con 0x7f147414be90 2024-03-25T18:16:37.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.015+0000 7f14717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.015+0000 7f14717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.015+0000 7f14717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.015+0000 7f14717fa640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:16:37.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.015+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1454004eb0 con 0x7f147414be90 2024-03-25T18:16:37.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.016+0000 7f14717fa640 10 client.?.objecter ms_handle_connect 0x7f14740b4d60 2024-03-25T18:16:37.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.016+0000 7f14717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.020+0000 7f14717fa640 10 client.?.objecter ms_handle_connect 0x7f14740b1730 2024-03-25T18:16:37.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.020+0000 7f14717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.025+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2944245166 0 0) 0x7f1454004eb0 con 0x7f147414be90 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.025+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1474140380 con 0x7f147414be90 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.026+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f146c002060 con 0x7f147414be90 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.026+0000 7f14717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.026+0000 7f14717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.026+0000 7f14717fa640 10 monclient(hunting): dump: 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:37.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.026+0000 7f14717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:37.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.034+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942908513 0 0) 0x7f14741406f0 con 0x7f14740b1730 2024-03-25T18:16:37.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.034+0000 7f14717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.034+0000 7f14717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.034+0000 7f14717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.034+0000 7f14717fa640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:16:37.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.034+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1454004eb0 con 0x7f14740b1730 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3604365852 0 0) 0x7f1474140380 con 0x7f147414be90 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 >> v1:172.21.15.162:6790/0 conn(0x7f14740b4d60 legacy=0x7f14741d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 >> v1:172.21.15.162:6789/0 conn(0x7f14740b1730 legacy=0x7f1474131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 1 monclient: found mon.b 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1474132520 con 0x7f147414be90 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f14717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:37.045 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f147a28f640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:16:37.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f147a28f640 10 monclient: _renew_subs 2024-03-25T18:16:37.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f147a28f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:37.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.043+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1474132770 con 0x7f147414be90 2024-03-25T18:16:37.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:37.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 10 monclient: _renew_subs 2024-03-25T18:16:37.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:37.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f147413e130 con 0x7f147414be90 2024-03-25T18:16:37.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 1 librados: init done 2024-03-25T18:16:37.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:37.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:37.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:37.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.044+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1474140880 con 0x7f147414be90 2024-03-25T18:16:37.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.066+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f146c003ce0 con 0x7f147414be90 2024-03-25T18:16:37.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.066+0000 7f14717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:37.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.066+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f146c0040b0 con 0x7f147414be90 2024-03-25T18:16:37.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.066+0000 7f14717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.066+0000 7f14717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:37.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.066+0000 7f14717fa640 10 monclient: dump: 2024-03-25T18:16:37.067 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:37.068 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.066+0000 7f1472ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:37.069 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.067+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f146c0042d0 con 0x7f147414be90 2024-03-25T18:16:37.069 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.068+0000 7f1473fff640 10 monclient: get_auth_request con 0x7f145406dd40 auth_method 0 2024-03-25T18:16:37.069 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.068+0000 7f14717fa640 10 client.4327.objecter ms_handle_connect 0x7f145406dd40 2024-03-25T18:16:37.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.070+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f146c054bc0 con 0x7f147414be90 2024-03-25T18:16:37.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.070+0000 7f14717fa640 10 client.4327.objecter ms_dispatch 0x7f1474000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:37.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.070+0000 7f14717fa640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:37.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.070+0000 7f14717fa640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:37.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.070+0000 7f14717fa640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-25T18:16:37.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.331+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f146c0506d0 con 0x7f147414be90 2024-03-25T18:16:37.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.331+0000 7f14717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:37.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.331+0000 7f14717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:37.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.582+0000 7f147a28f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:37.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.582+0000 7f147a28f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:37.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.582+0000 7f147a28f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:16:37.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.582+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1474140510 con 0x7f147414be90 2024-03-25T18:16:37.681 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.680+0000 7f14717fa640 1 -- 172.21.15.162:0/772612679 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f146c052600 con 0x7f147414be90 2024-03-25T18:16:37.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.681+0000 7f14717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:37.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.681+0000 7f14717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:37.682 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:16:37.682 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:37.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.683+0000 7f147a28f640 10 librados: watch_flush enter 2024-03-25T18:16:37.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 10 librados: watch_flush exit 2024-03-25T18:16:37.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-25T18:16:37.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 10 client.4327.objecter successfully canceled tick 2024-03-25T18:16:37.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 >> v1:172.21.15.191:6800/1552703795 conn(0x7f145406dd40 legacy=0x7f1454070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 10 monclient: shutdown 2024-03-25T18:16:37.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:37.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 >> v1:172.21.15.191:6789/0 conn(0x7f147414be90 legacy=0x7f14741d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 shutdown_connections 2024-03-25T18:16:37.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 >> 172.21.15.162:0/772612679 conn(0x7f14740617a0 msgr2=0x7f147413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:37.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 shutdown_connections 2024-03-25T18:16:37.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 1 -- 172.21.15.162:0/772612679 wait complete. 2024-03-25T18:16:37.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:37.684+0000 7f147a28f640 1 librados: shutdown 2024-03-25T18:16:37.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:42.324 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:54.552 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.335 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:54.562 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.384 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:02:54.612 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.392 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:54.608 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.393 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:54.621 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.397 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:54.614 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.399 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:54.626 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.422 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:02:54.638 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:42.429 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:02:54.657 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:44.503 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:44.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.602+0000 7fd47fc4e640 10 monclient: get_monmap_and_config 2024-03-25T18:16:44.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.602+0000 7fd47fc4e640 10 monclient: build_initial_monmap 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.603+0000 7fd47fc4e640 10 monclient: monmap: 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:16:44.604021+0000 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:16:44.604021+0000 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:16:44.604 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:44.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.603+0000 7fd47fc4e640 1 Processor -- start 2024-03-25T18:16:44.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.603+0000 7fd47fc4e640 1 -- start start 2024-03-25T18:16:44.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.603+0000 7fd47fc4e640 10 monclient: init 2024-03-25T18:16:44.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:44.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:44.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd4780b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:44.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd4780b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:44.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd47814be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:44.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4780af940 con 0x7fd4780b4d60 2024-03-25T18:16:44.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4780b0a70 con 0x7fd47814be90 2024-03-25T18:16:44.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4780b0d50 con 0x7fd4780b1730 2024-03-25T18:16:44.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:44.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.604+0000 7fd47fc4e640 10 monclient(hunting): authenticate will time out at 7703.878299s 2024-03-25T18:16:44.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.616+0000 7fd47d9ea640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fd4780b4d60 0x7fd47813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39664/0 (socket says 172.21.15.162:39664) 2024-03-25T18:16:44.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.616+0000 7fd47d9ea640 1 -- 172.21.15.162:0/4235548860 learned_addr learned my addr 172.21.15.162:0/4235548860 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:44.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.628+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082208244 0 0) 0x7fd4780af940 con 0x7fd4780b4d60 2024-03-25T18:16:44.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.628+0000 7fd47c9e8640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.628+0000 7fd47c9e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.628+0000 7fd47c9e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.628+0000 7fd47c9e8640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:16:44.629 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.628+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd464003520 con 0x7fd4780b4d60 2024-03-25T18:16:44.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3992725629 0 0) 0x7fd464003520 con 0x7fd4780b4d60 2024-03-25T18:16:44.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 >> v1:172.21.15.162:6790/0 conn(0x7fd4780b1730 legacy=0x7fd4780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:44.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 >> v1:172.21.15.191:6789/0 conn(0x7fd47814be90 legacy=0x7fd47814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:44.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 10 monclient: _finish_hunting 0 2024-03-25T18:16:44.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 1 monclient: found mon.noname-a 2024-03-25T18:16:44.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4780b1030 con 0x7fd4780b4d60 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 10 monclient: _finish_auth 0 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 10 monclient: _check_auth_tickets 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd474001690 con 0x7fd4780b4d60 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47fc4e640 5 monclient: authenticate success, global_id 4344 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47fc4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47c9e8640 10 monclient: dump: 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.643 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:44.644 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:44.644 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:44.644 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:44.644 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:44.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.641+0000 7fd47fc4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:44.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.654+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd474001860 con 0x7fd4780b4d60 2024-03-25T18:16:44.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.654+0000 7fd47c9e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:44.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.654+0000 7fd47c9e8640 1 -- 172.21.15.162:0/4235548860 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd474001cb0 con 0x7fd4780b4d60 2024-03-25T18:16:44.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.654+0000 7fd47fc4e640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:44.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.654+0000 7fd47fc4e640 4 set_mon_vals no callback set 2024-03-25T18:16:44.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.654+0000 7fd47c9e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.654+0000 7fd47c9e8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.654+0000 7fd47c9e8640 10 monclient: dump: 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:44.656 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:44.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.655+0000 7fd47fc4e640 10 monclient: shutdown 2024-03-25T18:16:44.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.655+0000 7fd47fc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:44.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.655+0000 7fd47fc4e640 1 -- 172.21.15.162:0/4235548860 >> v1:172.21.15.162:6789/0 conn(0x7fd4780b4d60 legacy=0x7fd47813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.655+0000 7fd47fc4e640 1 -- 172.21.15.162:0/4235548860 shutdown_connections 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.655+0000 7fd47fc4e640 1 -- 172.21.15.162:0/4235548860 >> 172.21.15.162:0/4235548860 conn(0x7fd4780617a0 msgr2=0x7fd478061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.655+0000 7fd47fc4e640 1 -- 172.21.15.162:0/4235548860 shutdown_connections 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.655+0000 7fd47fc4e640 1 -- 172.21.15.162:0/4235548860 wait complete. 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 10 monclient: build_initial_monmap 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 10 monclient: monmap: 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:16:44.659 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 1 librados: starting msgr at 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 1 librados: starting objecter 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 1 Processor -- start 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 1 -- start start 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 1 librados: setting wanted keys 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 1 librados: calling monclient init 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.656+0000 7fd47fc4e640 10 monclient: init 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 10 monclient(hunting): picked mon.noname-a con 0x7fd4780b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 10 monclient(hunting): picked mon.noname-c con 0x7fd4780b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 10 monclient(hunting): picked mon.noname-b con 0x7fd47814be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4781385c0 con 0x7fd4780b1730 2024-03-25T18:16:44.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd478135f60 con 0x7fd47814be90 2024-03-25T18:16:44.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4781360f0 con 0x7fd4780b4d60 2024-03-25T18:16:44.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:44.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.657+0000 7fd47fc4e640 10 monclient(hunting): authenticate will time out at 7703.930854s 2024-03-25T18:16:44.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.661+0000 7fd47d9ea640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fd4780b1730 0x7fd4781d7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39708/0 (socket says 172.21.15.162:39708) 2024-03-25T18:16:44.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.661+0000 7fd47d9ea640 1 -- 172.21.15.162:0/2025980171 learned_addr learned my addr 172.21.15.162:0/2025980171 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:44.665 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.664+0000 7fd46e7fc640 10 client.?.objecter ms_handle_connect 0x7fd4780b1730 2024-03-25T18:16:44.665 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.664+0000 7fd46e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.673+0000 7fd46e7fc640 10 client.?.objecter ms_handle_connect 0x7fd4780b4d60 2024-03-25T18:16:44.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.673+0000 7fd46e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.673+0000 7fd46e7fc640 10 client.?.objecter ms_handle_connect 0x7fd47814be90 2024-03-25T18:16:44.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.673+0000 7fd46e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.676+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058481895 0 0) 0x7fd4781385c0 con 0x7fd4780b1730 2024-03-25T18:16:44.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.676+0000 7fd46e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.676+0000 7fd46e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.676+0000 7fd46e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.676+0000 7fd46e7fc640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:16:44.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.676+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd45c004eb0 con 0x7fd4780b1730 2024-03-25T18:16:44.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.683+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709129545 0 0) 0x7fd4781360f0 con 0x7fd4780b4d60 2024-03-25T18:16:44.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.683+0000 7fd46e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.683+0000 7fd46e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.683+0000 7fd46e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.683+0000 7fd46e7fc640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:16:44.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.683+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4781385c0 con 0x7fd4780b4d60 2024-03-25T18:16:44.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.687+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646052675 0 0) 0x7fd478135f60 con 0x7fd47814be90 2024-03-25T18:16:44.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.687+0000 7fd46e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.687+0000 7fd46e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.687+0000 7fd46e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.687+0000 7fd46e7fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:16:44.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.687+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4781360f0 con 0x7fd47814be90 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.697+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 585948084 0 0) 0x7fd45c004eb0 con 0x7fd4780b1730 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.697+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd478135f60 con 0x7fd4780b1730 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.697+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd474001b70 con 0x7fd4780b1730 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.697+0000 7fd46e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.697+0000 7fd46e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.697+0000 7fd46e7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:44.698 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:44.699 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:44.699 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.699 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:44.699 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:44.699 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:44.699 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:44.699 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:44.699 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.697+0000 7fd46e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.700+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2022569005 0 0) 0x7fd4781385c0 con 0x7fd4780b4d60 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.700+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd45c004eb0 con 0x7fd4780b4d60 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.700+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd468001810 con 0x7fd4780b4d60 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.700+0000 7fd46e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.700+0000 7fd46e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.700+0000 7fd46e7fc640 10 monclient(hunting): dump: 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:44.701 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:44.702 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.702 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:44.702 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:44.702 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:44.702 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:44.702 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2148698841 0 0) 0x7fd45c004eb0 con 0x7fd4780b4d60 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 >> v1:172.21.15.191:6789/0 conn(0x7fd47814be90 legacy=0x7fd4781e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 >> v1:172.21.15.162:6789/0 conn(0x7fd4780b1730 legacy=0x7fd4781d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 1 monclient: found mon.c 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478143900 con 0x7fd4780b4d60 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:44.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd46e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 5 monclient: authenticate success, global_id 4334 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 10 monclient: _renew_subs 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4781d82b0 con 0x7fd4780b4d60 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 10 monclient: _renew_subs 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4781307e0 con 0x7fd4780b4d60 2024-03-25T18:16:44.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.708+0000 7fd47fc4e640 1 librados: init done 2024-03-25T18:16:44.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.710+0000 7fd47fc4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:44.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.710+0000 7fd47fc4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:44.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.710+0000 7fd47fc4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:44.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:44.710+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4780b22f0 con 0x7fd4780b4d60 2024-03-25T18:16:45.004 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.003+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd468001d00 con 0x7fd4780b4d60 2024-03-25T18:16:45.004 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.003+0000 7fd46e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:45.004 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.003+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd4680035f0 con 0x7fd4780b4d60 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.003+0000 7fd46e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.003+0000 7fd46e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.003+0000 7fd46e7fc640 10 monclient: dump: 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:45.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.003+0000 7fd46ffff640 4 set_mon_vals no callback set 2024-03-25T18:16:45.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.004+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fd4680195c0 con 0x7fd4780b4d60 2024-03-25T18:16:45.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.005+0000 7fd47d9ea640 10 monclient: get_auth_request con 0x7fd45c06df10 auth_method 0 2024-03-25T18:16:45.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.005+0000 7fd46e7fc640 10 client.4334.objecter ms_handle_connect 0x7fd45c06df10 2024-03-25T18:16:45.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.016+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fd4680533c0 con 0x7fd4780b4d60 2024-03-25T18:16:45.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.016+0000 7fd46e7fc640 10 client.4334.objecter ms_dispatch 0x7fd478000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:45.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.016+0000 7fd46e7fc640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:45.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.016+0000 7fd46e7fc640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:45.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.016+0000 7fd46e7fc640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-25T18:16:45.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.016+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd46804ed00 con 0x7fd4780b4d60 2024-03-25T18:16:45.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.016+0000 7fd46e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:45.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.016+0000 7fd46e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:45.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.261+0000 7fd47fc4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:45.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.261+0000 7fd47fc4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:45.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.261+0000 7fd47fc4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:45.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.261+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4781383c0 con 0x7fd4780b4d60 2024-03-25T18:16:45.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.287+0000 7fd46e7fc640 1 -- 172.21.15.162:0/2025980171 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fd468050e00 con 0x7fd4780b4d60 2024-03-25T18:16:45.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.287+0000 7fd46e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:45.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.287+0000 7fd46e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:45.289 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:16:45.289 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:45.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 10 librados: watch_flush enter 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 10 librados: watch_flush exit 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 10 client.4334.objecter successfully canceled tick 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 >> v1:172.21.15.191:6800/1552703795 conn(0x7fd45c06df10 legacy=0x7fd45c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 10 monclient: shutdown 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 >> v1:172.21.15.162:6790/0 conn(0x7fd4780b4d60 legacy=0x7fd47812fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 shutdown_connections 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 >> 172.21.15.162:0/2025980171 conn(0x7fd4780617a0 msgr2=0x7fd4780b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 shutdown_connections 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 1 -- 172.21.15.162:0/2025980171 wait complete. 2024-03-25T18:16:45.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:45.290+0000 7fd47fc4e640 1 librados: shutdown 2024-03-25T18:16:45.303 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:52.109 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:52.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.226+0000 7f7d55f5d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:52.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.226+0000 7f7d55f5d640 10 monclient: build_initial_monmap 2024-03-25T18:16:52.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.226+0000 7f7d55f5d640 10 monclient: monmap: 2024-03-25T18:16:52.227 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:52.227 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:52.227 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:16:52.227469+0000 2024-03-25T18:16:52.228 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:16:52.227469+0000 2024-03-25T18:16:52.228 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:52.228 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:52.228 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:52.228 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:16:52.228 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:16:52.228 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:52.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.227+0000 7f7d55f5d640 1 Processor -- start 2024-03-25T18:16:52.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.227+0000 7f7d55f5d640 1 -- start start 2024-03-25T18:16:52.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.227+0000 7f7d55f5d640 10 monclient: init 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d500b1ef0 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d501473a0 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d50150070 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d500b2570 con 0x7f7d500b1ef0 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d50146c20 con 0x7f7d501473a0 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d50146e80 con 0x7f7d50150070 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:52.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.228+0000 7f7d55f5d640 10 monclient(hunting): authenticate will time out at 7711.502355s 2024-03-25T18:16:52.238 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.237+0000 7f7d4effd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f7d501473a0 0x7f7d50143b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:34360/0 (socket says 172.21.15.162:34360) 2024-03-25T18:16:52.238 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.237+0000 7f7d4effd640 1 -- 172.21.15.162:0/2560802263 learned_addr learned my addr 172.21.15.162:0/2560802263 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:52.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.253+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862592085 0 0) 0x7f7d50146c20 con 0x7f7d501473a0 2024-03-25T18:16:52.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.253+0000 7f7d4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.253+0000 7f7d4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.253+0000 7f7d4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.253+0000 7f7d4e7fc640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:16:52.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.253+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d38003520 con 0x7f7d501473a0 2024-03-25T18:16:52.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.255+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236697653 0 0) 0x7f7d50146e80 con 0x7f7d50150070 2024-03-25T18:16:52.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.255+0000 7f7d4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.255+0000 7f7d4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.255+0000 7f7d4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.255+0000 7f7d4e7fc640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:16:52.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.255+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d50146c20 con 0x7f7d50150070 2024-03-25T18:16:52.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.256+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194610657 0 0) 0x7f7d500b2570 con 0x7f7d500b1ef0 2024-03-25T18:16:52.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.256+0000 7f7d4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.256+0000 7f7d4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.256+0000 7f7d4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.256+0000 7f7d4e7fc640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:16:52.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.256+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d50146e80 con 0x7f7d500b1ef0 2024-03-25T18:16:52.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.277+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2227338422 0 0) 0x7f7d50146c20 con 0x7f7d50150070 2024-03-25T18:16:52.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.277+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 >> v1:172.21.15.191:6789/0 conn(0x7f7d501473a0 legacy=0x7f7d50143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.277+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 >> v1:172.21.15.162:6789/0 conn(0x7f7d500b1ef0 legacy=0x7f7d500b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:52.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:16:52.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:52.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:16:52.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5006e1b0 con 0x7f7d50150070 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d55f5d640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d55f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7d480019f0 con 0x7f7d50150070 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d4e7fc640 10 monclient: dump: 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:52.280 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:52.281 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:52.281 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:52.281 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:52.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.278+0000 7f7d55f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:52.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.305+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d48001b50 con 0x7f7d50150070 2024-03-25T18:16:52.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.305+0000 7f7d4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.305+0000 7f7d4e7fc640 1 -- 172.21.15.162:0/2560802263 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7d48001f60 con 0x7f7d50150070 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.305+0000 7f7d55f5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.305+0000 7f7d55f5d640 4 set_mon_vals no callback set 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.305+0000 7f7d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.305+0000 7f7d4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.305+0000 7f7d4e7fc640 10 monclient: dump: 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:52.307 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:52.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.306+0000 7f7d55f5d640 10 monclient: shutdown 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.306+0000 7f7d55f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.306+0000 7f7d55f5d640 1 -- 172.21.15.162:0/2560802263 >> v1:172.21.15.162:6790/0 conn(0x7f7d50150070 legacy=0x7f7d50152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.307+0000 7f7d55f5d640 1 -- 172.21.15.162:0/2560802263 shutdown_connections 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.307+0000 7f7d55f5d640 1 -- 172.21.15.162:0/2560802263 >> 172.21.15.162:0/2560802263 conn(0x7f7d500b7ea0 msgr2=0x7f7d500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.307+0000 7f7d55f5d640 1 -- 172.21.15.162:0/2560802263 shutdown_connections 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.307+0000 7f7d55f5d640 1 -- 172.21.15.162:0/2560802263 wait complete. 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.307+0000 7f7d55f5d640 10 monclient: build_initial_monmap 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.307+0000 7f7d55f5d640 10 monclient: monmap: 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:52.309 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:16:52.310 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:16:52.310 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:52.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.308+0000 7f7d55f5d640 1 librados: starting msgr at 2024-03-25T18:16:52.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.308+0000 7f7d55f5d640 1 librados: starting objecter 2024-03-25T18:16:52.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.308+0000 7f7d55f5d640 1 Processor -- start 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.308+0000 7f7d55f5d640 1 -- start start 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.308+0000 7f7d55f5d640 1 librados: setting wanted keys 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.308+0000 7f7d55f5d640 1 librados: calling monclient init 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.308+0000 7f7d55f5d640 10 monclient: init 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d500b1ef0 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d501473a0 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d50150070 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d50144560 con 0x7f7d501473a0 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d501dbb20 con 0x7f7d500b1ef0 2024-03-25T18:16:52.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d501455e0 con 0x7f7d50150070 2024-03-25T18:16:52.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:52.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.309+0000 7f7d55f5d640 10 monclient(hunting): authenticate will time out at 7711.583047s 2024-03-25T18:16:52.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.314+0000 7f7d4ffff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f7d50150070 0x7f7d501e64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:50554/0 (socket says 172.21.15.162:50554) 2024-03-25T18:16:52.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.314+0000 7f7d4ffff640 1 -- 172.21.15.162:0/1047908774 learned_addr learned my addr 172.21.15.162:0/1047908774 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:52.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.316+0000 7f7d37fff640 10 client.?.objecter ms_handle_connect 0x7f7d50150070 2024-03-25T18:16:52.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.316+0000 7f7d37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:52.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.321+0000 7f7d37fff640 10 client.?.objecter ms_handle_connect 0x7f7d501473a0 2024-03-25T18:16:52.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.321+0000 7f7d37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:52.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.332+0000 7f7d37fff640 10 client.?.objecter ms_handle_connect 0x7f7d500b1ef0 2024-03-25T18:16:52.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.332+0000 7f7d37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:52.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.332+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525949565 0 0) 0x7f7d501455e0 con 0x7f7d50150070 2024-03-25T18:16:52.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.332+0000 7f7d37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.332+0000 7f7d37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.332+0000 7f7d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.332+0000 7f7d37fff640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:16:52.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.332+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d3c004eb0 con 0x7f7d50150070 2024-03-25T18:16:52.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.336+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961031719 0 0) 0x7f7d50144560 con 0x7f7d501473a0 2024-03-25T18:16:52.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.336+0000 7f7d37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.336+0000 7f7d37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.336+0000 7f7d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.336+0000 7f7d37fff640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:16:52.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.336+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d501455e0 con 0x7f7d501473a0 2024-03-25T18:16:52.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.344+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353661896 0 0) 0x7f7d501dbb20 con 0x7f7d500b1ef0 2024-03-25T18:16:52.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.344+0000 7f7d37fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.344+0000 7f7d37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.344+0000 7f7d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.344+0000 7f7d37fff640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:16:52.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.344+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d50144560 con 0x7f7d500b1ef0 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.348+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 510437856 0 0) 0x7f7d3c004eb0 con 0x7f7d50150070 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.348+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d501dbb20 con 0x7f7d50150070 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.348+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7d48001f20 con 0x7f7d50150070 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.348+0000 7f7d37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.348+0000 7f7d37fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.348+0000 7f7d37fff640 10 monclient(hunting): dump: 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.349 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.350 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:52.350 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:52.350 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:52.350 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:52.350 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:52.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.348+0000 7f7d37fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:52.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.365+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 855325225 0 0) 0x7f7d501455e0 con 0x7f7d501473a0 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.365+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d3c004eb0 con 0x7f7d501473a0 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7d44001950 con 0x7f7d501473a0 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 10 monclient(hunting): dump: 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:52.367 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2180461588 0 0) 0x7f7d50144560 con 0x7f7d500b1ef0 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d501455e0 con 0x7f7d500b1ef0 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7d40001870 con 0x7f7d500b1ef0 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.366+0000 7f7d37fff640 10 monclient(hunting): dump: 2024-03-25T18:16:52.368 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:52.369 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:52.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 103611278 0 0) 0x7f7d501dbb20 con 0x7f7d50150070 2024-03-25T18:16:52.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 >> v1:172.21.15.191:6789/0 conn(0x7f7d500b1ef0 legacy=0x7f7d501dc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 >> v1:172.21.15.162:6789/0 conn(0x7f7d501473a0 legacy=0x7f7d50070800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 1 monclient: found mon.c 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d50145770 con 0x7f7d50150070 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 10 monclient: _renew_subs 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d501dc730 con 0x7f7d50150070 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 10 monclient: _renew_subs 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:52.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d500710f0 con 0x7f7d50150070 2024-03-25T18:16:52.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.368+0000 7f7d55f5d640 1 librados: init done 2024-03-25T18:16:52.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.369+0000 7f7d55f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:52.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.369+0000 7f7d55f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:52.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.369+0000 7f7d55f5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:52.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.369+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d501dbcb0 con 0x7f7d50150070 2024-03-25T18:16:52.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.675+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d48003c20 con 0x7f7d50150070 2024-03-25T18:16:52.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.675+0000 7f7d37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:52.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.675+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7d48003e90 con 0x7f7d50150070 2024-03-25T18:16:52.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.675+0000 7f7d37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.675+0000 7f7d37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.675+0000 7f7d37fff640 10 monclient: dump: 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:52.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.675+0000 7f7d4dffb640 4 set_mon_vals no callback set 2024-03-25T18:16:52.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.676+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f7d480040b0 con 0x7f7d50150070 2024-03-25T18:16:52.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.677+0000 7f7d4f7fe640 10 monclient: get_auth_request con 0x7f7d3c06e0f0 auth_method 0 2024-03-25T18:16:52.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.677+0000 7f7d37fff640 10 client.4340.objecter ms_handle_connect 0x7f7d3c06e0f0 2024-03-25T18:16:52.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.681+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f7d480549e0 con 0x7f7d50150070 2024-03-25T18:16:52.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.681+0000 7f7d37fff640 10 client.4340.objecter ms_dispatch 0x7f7d50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:52.683 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.681+0000 7f7d37fff640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:52.683 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.681+0000 7f7d37fff640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:52.683 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.681+0000 7f7d37fff640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T18:16:52.683 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.681+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d480504f0 con 0x7f7d50150070 2024-03-25T18:16:52.683 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.681+0000 7f7d37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:52.683 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.681+0000 7f7d37fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:52.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.912+0000 7f7d55f5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:52.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.912+0000 7f7d55f5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:52.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.912+0000 7f7d55f5d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:16:52.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.912+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d500627b0 con 0x7f7d50150070 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.983+0000 7f7d37fff640 1 -- 172.21.15.162:0/1047908774 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f7d48052420 con 0x7f7d50150070 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.983+0000 7f7d37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.983+0000 7f7d37fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:16:52.985 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.985+0000 7f7d55f5d640 10 librados: watch_flush enter 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 10 librados: watch_flush exit 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 10 client.4340.objecter successfully canceled tick 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 >> v1:172.21.15.191:6800/1552703795 conn(0x7f7d3c06e0f0 legacy=0x7f7d3c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 10 monclient: shutdown 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 >> v1:172.21.15.162:6790/0 conn(0x7f7d50150070 legacy=0x7f7d501e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 shutdown_connections 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 >> 172.21.15.162:0/1047908774 conn(0x7f7d500b7ea0 msgr2=0x7f7d501416c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 shutdown_connections 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 1 -- 172.21.15.162:0/1047908774 wait complete. 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:52.986+0000 7f7d55f5d640 1 librados: shutdown 2024-03-25T18:16:52.999 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:59.806 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.920+0000 7f0eb0e45640 10 monclient: get_monmap_and_config 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.920+0000 7f0eb0e45640 10 monclient: build_initial_monmap 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.920+0000 7f0eb0e45640 10 monclient: monmap: 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:16:59.921958+0000 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:16:59.921958+0000 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:59.922 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:16:59.923 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:16:59.923 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.921+0000 7f0eb0e45640 1 Processor -- start 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.921+0000 7f0eb0e45640 1 -- start start 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.921+0000 7f0eb0e45640 10 monclient: init 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 10 monclient(hunting): picked mon.noname-a con 0x7f0eac0a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 10 monclient(hunting): picked mon.noname-c con 0x7f0eac13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 10 monclient(hunting): picked mon.noname-b con 0x7f0eac1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac142520 con 0x7f0eac0a98f0 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac0b28a0 con 0x7f0eac13f710 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac0b2a30 con 0x7f0eac1430c0 2024-03-25T18:16:59.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:59.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.922+0000 7f0eb0e45640 10 monclient(hunting): authenticate will time out at 7719.196535s 2024-03-25T18:16:59.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.931+0000 7f0eaad9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f0eac1430c0 0x7f0eac14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:57956/0 (socket says 172.21.15.162:57956) 2024-03-25T18:16:59.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.931+0000 7f0eaad9d640 1 -- 172.21.15.162:0/3648931443 learned_addr learned my addr 172.21.15.162:0/3648931443 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:59.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.943+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135400608 0 0) 0x7f0eac0b2a30 con 0x7f0eac1430c0 2024-03-25T18:16:59.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.943+0000 7f0ea959a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.943+0000 7f0ea959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.943+0000 7f0ea959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.943+0000 7f0ea959a640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:16:59.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.943+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e98003520 con 0x7f0eac1430c0 2024-03-25T18:16:59.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.964+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403580826 0 0) 0x7f0eac142520 con 0x7f0eac0a98f0 2024-03-25T18:16:59.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.964+0000 7f0ea959a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.964+0000 7f0ea959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.964+0000 7f0ea959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.964+0000 7f0ea959a640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:16:59.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.964+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eac0b2a30 con 0x7f0eac0a98f0 2024-03-25T18:16:59.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.965+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273419982 0 0) 0x7f0eac0b28a0 con 0x7f0eac13f710 2024-03-25T18:16:59.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.965+0000 7f0ea959a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.965+0000 7f0ea959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.965+0000 7f0ea959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.965+0000 7f0ea959a640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:16:59.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.965+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eac142520 con 0x7f0eac13f710 2024-03-25T18:16:59.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3113336320 0 0) 0x7f0e98003520 con 0x7f0eac1430c0 2024-03-25T18:16:59.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 >> v1:172.21.15.191:6789/0 conn(0x7f0eac13f710 legacy=0x7f0eac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 >> v1:172.21.15.162:6789/0 conn(0x7f0eac0a98f0 legacy=0x7f0eac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 10 monclient: _finish_hunting 0 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 1 monclient: found mon.noname-b 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0b44c0 con 0x7f0eac1430c0 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 10 monclient: _finish_auth 0 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 10 monclient: _check_auth_tickets 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0ea4001a10 con 0x7f0eac1430c0 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0ea959a640 10 monclient: dump: 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:59.968 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0eb0e45640 5 monclient: authenticate success, global_id 4343 2024-03-25T18:16:59.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.966+0000 7f0eb0e45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:59.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.974+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ea4001b70 con 0x7f0eac1430c0 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.974+0000 7f0ea959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.974+0000 7f0ea959a640 1 -- 172.21.15.162:0/3648931443 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0ea4001f80 con 0x7f0eac1430c0 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.974+0000 7f0eb0e45640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.974+0000 7f0eb0e45640 4 set_mon_vals no callback set 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.974+0000 7f0ea959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.974+0000 7f0ea959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.974+0000 7f0ea959a640 10 monclient: dump: 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:16:59.976 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:16:59.977 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:16:59.977 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:59.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.975+0000 7f0eb0e45640 10 monclient: shutdown 2024-03-25T18:16:59.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.975+0000 7f0eb0e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:59.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.975+0000 7f0eb0e45640 1 -- 172.21.15.162:0/3648931443 >> v1:172.21.15.162:6790/0 conn(0x7f0eac1430c0 legacy=0x7f0eac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.975+0000 7f0eb0e45640 1 -- 172.21.15.162:0/3648931443 shutdown_connections 2024-03-25T18:16:59.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.975+0000 7f0eb0e45640 1 -- 172.21.15.162:0/3648931443 >> 172.21.15.162:0/3648931443 conn(0x7f0eac0b7ea0 msgr2=0x7f0eac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:59.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.975+0000 7f0eb0e45640 1 -- 172.21.15.162:0/3648931443 shutdown_connections 2024-03-25T18:16:59.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.976+0000 7f0eb0e45640 1 -- 172.21.15.162:0/3648931443 wait complete. 2024-03-25T18:16:59.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.976+0000 7f0eb0e45640 10 monclient: build_initial_monmap 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.976+0000 7f0eb0e45640 10 monclient: monmap: 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 1 librados: starting msgr at 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 1 librados: starting objecter 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 1 Processor -- start 2024-03-25T18:16:59.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 1 -- start start 2024-03-25T18:16:59.980 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 1 librados: setting wanted keys 2024-03-25T18:16:59.980 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 1 librados: calling monclient init 2024-03-25T18:16:59.980 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 10 monclient: init 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 10 monclient(hunting): picked mon.noname-b con 0x7f0eac0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.977+0000 7f0eb0e45640 10 monclient(hunting): picked mon.noname-a con 0x7f0eac13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.978+0000 7f0eb0e45640 10 monclient(hunting): picked mon.noname-c con 0x7f0eac1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.978+0000 7f0eb0e45640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac136ad0 con 0x7f0eac13f710 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.978+0000 7f0eb0e45640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac0b7520 con 0x7f0eac0a98f0 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.978+0000 7f0eb0e45640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eac135810 con 0x7f0eac1430c0 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.978+0000 7f0eb0e45640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:59.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.978+0000 7f0eb0e45640 10 monclient(hunting): authenticate will time out at 7719.251809s 2024-03-25T18:16:59.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.991+0000 7f0eaa59c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f0eac0a98f0 0x7f0eac135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:59488/0 (socket says 172.21.15.162:59488) 2024-03-25T18:16:59.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.991+0000 7f0eaa59c640 1 -- 172.21.15.162:0/2886097715 learned_addr learned my addr 172.21.15.162:0/2886097715 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:16:59.994 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.993+0000 7f0ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f0eac1430c0 2024-03-25T18:16:59.994 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:16:59.993+0000 7f0ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.000+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964309818 0 0) 0x7f0eac135810 con 0x7f0eac1430c0 2024-03-25T18:17:00.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.000+0000 7f0ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.000+0000 7f0ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.000+0000 7f0ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.000+0000 7f0ea2ffd640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:17:00.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.000+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e8c004eb0 con 0x7f0eac1430c0 2024-03-25T18:17:00.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.000+0000 7f0ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f0eac13f710 2024-03-25T18:17:00.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.000+0000 7f0ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.001+0000 7f0ea2ffd640 10 client.?.objecter ms_handle_connect 0x7f0eac0a98f0 2024-03-25T18:17:00.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.001+0000 7f0ea2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.010+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1718688943 0 0) 0x7f0e8c004eb0 con 0x7f0eac1430c0 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.011+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0eac135810 con 0x7f0eac1430c0 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.011+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0ea4001e80 con 0x7f0eac1430c0 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.011+0000 7f0ea2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.011+0000 7f0ea2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.011+0000 7f0ea2ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:00.012 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.011+0000 7f0ea2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:00.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.014+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060859989 0 0) 0x7f0eac0b7520 con 0x7f0eac0a98f0 2024-03-25T18:17:00.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.014+0000 7f0ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.014+0000 7f0ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.014+0000 7f0ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.014+0000 7f0ea2ffd640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:17:00.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.014+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e8c004eb0 con 0x7f0eac0a98f0 2024-03-25T18:17:00.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.021+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732893404 0 0) 0x7f0eac136ad0 con 0x7f0eac13f710 2024-03-25T18:17:00.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.021+0000 7f0ea2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.021+0000 7f0ea2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.021+0000 7f0ea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.021+0000 7f0ea2ffd640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:17:00.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.021+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eac0b7520 con 0x7f0eac13f710 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2511292479 0 0) 0x7f0eac135810 con 0x7f0eac1430c0 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 >> v1:172.21.15.191:6789/0 conn(0x7f0eac0a98f0 legacy=0x7f0eac135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 >> v1:172.21.15.162:6789/0 conn(0x7f0eac13f710 legacy=0x7f0eac0b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 1 monclient: found mon.c 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eac0b6180 con 0x7f0eac1430c0 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0eb0e45640 5 monclient: authenticate success, global_id 4346 2024-03-25T18:17:00.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0eb0e45640 10 monclient: _renew_subs 2024-03-25T18:17:00.028 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0eb0e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:00.028 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac0b7a40 con 0x7f0eac1430c0 2024-03-25T18:17:00.028 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0eb0e45640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:00.028 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.025+0000 7f0eb0e45640 10 monclient: _renew_subs 2024-03-25T18:17:00.028 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.026+0000 7f0eb0e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:00.028 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.026+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac131840 con 0x7f0eac1430c0 2024-03-25T18:17:00.028 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.026+0000 7f0eb0e45640 1 librados: init done 2024-03-25T18:17:00.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.027+0000 7f0eb0e45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:00.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.027+0000 7f0eb0e45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:00.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.027+0000 7f0eb0e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:00.029 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.027+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eac136f60 con 0x7f0eac1430c0 2024-03-25T18:17:00.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.328+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ea4002080 con 0x7f0eac1430c0 2024-03-25T18:17:00.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.328+0000 7f0ea2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:00.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.328+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0ea4004e10 con 0x7f0eac1430c0 2024-03-25T18:17:00.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.328+0000 7f0ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.328+0000 7f0ea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.328+0000 7f0ea2ffd640 10 monclient: dump: 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:00.330 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.328+0000 7f0ea8d99640 4 set_mon_vals no callback set 2024-03-25T18:17:00.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.329+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f0ea401ade0 con 0x7f0eac1430c0 2024-03-25T18:17:00.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.330+0000 7f0eaa59c640 10 monclient: get_auth_request con 0x7f0e8c06de10 auth_method 0 2024-03-25T18:17:00.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.330+0000 7f0ea2ffd640 10 client.4346.objecter ms_handle_connect 0x7f0e8c06de10 2024-03-25T18:17:00.336 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.335+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f0ea4054990 con 0x7f0eac1430c0 2024-03-25T18:17:00.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.335+0000 7f0ea2ffd640 10 client.4346.objecter ms_dispatch 0x7f0eac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:00.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.335+0000 7f0ea2ffd640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:00.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.335+0000 7f0ea2ffd640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:00.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.335+0000 7f0ea2ffd640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-25T18:17:00.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.335+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ea40504a0 con 0x7f0eac1430c0 2024-03-25T18:17:00.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.335+0000 7f0ea2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:00.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.335+0000 7f0ea2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:00.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.579+0000 7f0eb0e45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:00.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.579+0000 7f0eb0e45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:00.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.579+0000 7f0eb0e45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:00.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.579+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0eac0b6310 con 0x7f0eac1430c0 2024-03-25T18:17:00.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.610+0000 7f0ea2ffd640 1 -- 172.21.15.162:0/2886097715 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f0ea40523d0 con 0x7f0eac1430c0 2024-03-25T18:17:00.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.610+0000 7f0ea2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:00.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.610+0000 7f0ea2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:00.611 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:17:00.611 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.612+0000 7f0eb0e45640 10 librados: watch_flush enter 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.612+0000 7f0eb0e45640 10 librados: watch_flush exit 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.612+0000 7f0eb0e45640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.612+0000 7f0eb0e45640 10 client.4346.objecter successfully canceled tick 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.612+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 >> v1:172.21.15.191:6800/1552703795 conn(0x7f0e8c06de10 legacy=0x7f0e8c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.612+0000 7f0eb0e45640 10 monclient: shutdown 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.612+0000 7f0eb0e45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.612+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 >> v1:172.21.15.162:6790/0 conn(0x7f0eac1430c0 legacy=0x7f0eac130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.613+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 shutdown_connections 2024-03-25T18:17:00.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.613+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 >> 172.21.15.162:0/2886097715 conn(0x7f0eac0b7ea0 msgr2=0x7f0eac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:00.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.613+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 shutdown_connections 2024-03-25T18:17:00.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.613+0000 7f0eb0e45640 1 -- 172.21.15.162:0/2886097715 wait complete. 2024-03-25T18:17:00.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:00.613+0000 7f0eb0e45640 1 librados: shutdown 2024-03-25T18:17:00.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:02.419 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:14.646 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.424 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:14.652 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.465 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:14.693 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.472 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:14.700 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.474 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:14.691 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.483 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:14.699 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.483 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:14.711 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.514 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:14.742 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:02.516 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:14.732 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:07.434 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:07.551 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.550+0000 7f82c460e640 10 monclient: get_monmap_and_config 2024-03-25T18:17:07.551 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.550+0000 7f82c460e640 10 monclient: build_initial_monmap 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.550+0000 7f82c460e640 10 monclient: monmap: 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:17:07.551986+0000 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:17:07.551986+0000 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:17:07.552 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:07.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.551+0000 7f82c460e640 1 Processor -- start 2024-03-25T18:17:07.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.551+0000 7f82c460e640 1 -- start start 2024-03-25T18:17:07.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.551+0000 7f82c460e640 10 monclient: init 2024-03-25T18:17:07.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 10 monclient(hunting): picked mon.noname-c con 0x7f82bc0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 10 monclient(hunting): picked mon.noname-a con 0x7f82bc13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 10 monclient(hunting): picked mon.noname-b con 0x7f82bc1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc142520 con 0x7f82bc13f710 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0b28a0 con 0x7f82bc0a98f0 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0b2a30 con 0x7f82bc1430c0 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:07.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.552+0000 7f82c460e640 10 monclient(hunting): authenticate will time out at 7726.826391s 2024-03-25T18:17:07.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.557+0000 7f82c2bab640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f82bc1430c0 0x7f82bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:40708/0 (socket says 172.21.15.162:40708) 2024-03-25T18:17:07.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.557+0000 7f82c2bab640 1 -- 172.21.15.162:0/3229832869 learned_addr learned my addr 172.21.15.162:0/3229832869 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:07.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.572+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122938820 0 0) 0x7f82bc0b2a30 con 0x7f82bc1430c0 2024-03-25T18:17:07.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.572+0000 7f82c13a8640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.572+0000 7f82c13a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.572+0000 7f82c13a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.574 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.572+0000 7f82c13a8640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:17:07.574 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.572+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a8003520 con 0x7f82bc1430c0 2024-03-25T18:17:07.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230511945 0 0) 0x7f82bc142520 con 0x7f82bc13f710 2024-03-25T18:17:07.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc0b2a30 con 0x7f82bc13f710 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811035373 0 0) 0x7f82bc0b28a0 con 0x7f82bc0a98f0 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:17:07.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.581+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc142520 con 0x7f82bc0a98f0 2024-03-25T18:17:07.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2458040531 0 0) 0x7f82a8003520 con 0x7f82bc1430c0 2024-03-25T18:17:07.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 >> v1:172.21.15.191:6789/0 conn(0x7f82bc0a98f0 legacy=0x7f82bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 >> v1:172.21.15.162:6789/0 conn(0x7f82bc13f710 legacy=0x7f82bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 10 monclient: _finish_hunting 0 2024-03-25T18:17:07.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 1 monclient: found mon.noname-b 2024-03-25T18:17:07.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:07.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc0b44c0 con 0x7f82bc1430c0 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 10 monclient: _finish_auth 0 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 10 monclient: _check_auth_tickets 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c13a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c460e640 5 monclient: authenticate success, global_id 4349 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.583+0000 7f82c460e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.584+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82b40019d0 con 0x7f82bc1430c0 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.584+0000 7f82c13a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.584+0000 7f82c13a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:07.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.584+0000 7f82c13a8640 10 monclient: dump: 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:07.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.584+0000 7f82c460e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:07.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.591+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b4001c20 con 0x7f82bc1430c0 2024-03-25T18:17:07.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.591+0000 7f82c13a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:07.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.591+0000 7f82c13a8640 1 -- 172.21.15.162:0/3229832869 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82b4001fb0 con 0x7f82bc1430c0 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.591+0000 7f82c460e640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.591+0000 7f82c460e640 4 set_mon_vals no callback set 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.591+0000 7f82c13a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.591+0000 7f82c13a8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.591+0000 7f82c13a8640 10 monclient: dump: 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:07.593 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:07.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.592+0000 7f82c460e640 10 monclient: shutdown 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.592+0000 7f82c460e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.592+0000 7f82c460e640 1 -- 172.21.15.162:0/3229832869 >> v1:172.21.15.162:6790/0 conn(0x7f82bc1430c0 legacy=0x7f82bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.592+0000 7f82c460e640 1 -- 172.21.15.162:0/3229832869 shutdown_connections 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.592+0000 7f82c460e640 1 -- 172.21.15.162:0/3229832869 >> 172.21.15.162:0/3229832869 conn(0x7f82bc0b7ea0 msgr2=0x7f82bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.592+0000 7f82c460e640 1 -- 172.21.15.162:0/3229832869 shutdown_connections 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.593+0000 7f82c460e640 1 -- 172.21.15.162:0/3229832869 wait complete. 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 10 monclient: build_initial_monmap 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 10 monclient: monmap: 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:07.595 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:17:07.596 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:17:07.596 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 1 librados: starting msgr at 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 1 librados: starting objecter 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 1 Processor -- start 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 1 -- start start 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 1 librados: setting wanted keys 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 1 librados: calling monclient init 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.594+0000 7f82c460e640 10 monclient: init 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 10 monclient(hunting): picked mon.noname-a con 0x7f82bc0a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 10 monclient(hunting): picked mon.noname-b con 0x7f82bc13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 10 monclient(hunting): picked mon.noname-c con 0x7f82bc1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:07.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc1d7670 con 0x7f82bc0a98f0 2024-03-25T18:17:07.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc1d84f0 con 0x7f82bc13f710 2024-03-25T18:17:07.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82bc0ab8b0 con 0x7f82bc1430c0 2024-03-25T18:17:07.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:07.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.595+0000 7f82c460e640 10 monclient(hunting): authenticate will time out at 7726.869305s 2024-03-25T18:17:07.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.602+0000 7f82c1ba9640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f82bc13f710 0x7f82bc1381e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:47730/0 (socket says 172.21.15.162:47730) 2024-03-25T18:17:07.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.602+0000 7f82c1ba9640 1 -- 172.21.15.162:0/4141727356 learned_addr learned my addr 172.21.15.162:0/4141727356 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:07.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.603+0000 7f82b2ffd640 10 client.?.objecter ms_handle_connect 0x7f82bc13f710 2024-03-25T18:17:07.604 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.603+0000 7f82b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:07.613 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.612+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121735054 0 0) 0x7f82bc1d84f0 con 0x7f82bc13f710 2024-03-25T18:17:07.613 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.612+0000 7f82b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.613 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.612+0000 7f82b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.612+0000 7f82b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.612+0000 7f82b2ffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:17:07.614 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.612+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a0004eb0 con 0x7f82bc13f710 2024-03-25T18:17:07.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.617+0000 7f82b2ffd640 10 client.?.objecter ms_handle_connect 0x7f82bc1430c0 2024-03-25T18:17:07.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.617+0000 7f82b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:07.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.618+0000 7f82b2ffd640 10 client.?.objecter ms_handle_connect 0x7f82bc0a98f0 2024-03-25T18:17:07.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.618+0000 7f82b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:07.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.623+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268209824 0 0) 0x7f82bc0ab8b0 con 0x7f82bc1430c0 2024-03-25T18:17:07.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.623+0000 7f82b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.623+0000 7f82b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.623+0000 7f82b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.623+0000 7f82b2ffd640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:17:07.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.623+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82bc1d84f0 con 0x7f82bc1430c0 2024-03-25T18:17:07.626 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1074830306 0 0) 0x7f82a0004eb0 con 0x7f82bc13f710 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc0ab8b0 con 0x7f82bc13f710 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82ac0019b0 con 0x7f82bc13f710 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:07.627 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:07.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:07.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707069838 0 0) 0x7f82bc1d7670 con 0x7f82bc0a98f0 2024-03-25T18:17:07.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:07.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:07.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:07.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:17:07.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.625+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a0004eb0 con 0x7f82bc0a98f0 2024-03-25T18:17:07.639 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136751626 0 0) 0x7f82bc1d84f0 con 0x7f82bc1430c0 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc1d7670 con 0x7f82bc1430c0 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82b4001f30 con 0x7f82bc1430c0 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:07.640 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3548409422 0 0) 0x7f82a0004eb0 con 0x7f82bc0a98f0 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82bc1d84f0 con 0x7f82bc0a98f0 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82b80018f0 con 0x7f82bc0a98f0 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.639+0000 7f82b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.641 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.642 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:07.642 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:07.642 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:07.642 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:07.642 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1900268414 0 0) 0x7f82bc0ab8b0 con 0x7f82bc13f710 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 >> v1:172.21.15.162:6790/0 conn(0x7f82bc1430c0 legacy=0x7f82bc1e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 >> v1:172.21.15.162:6789/0 conn(0x7f82bc0a98f0 legacy=0x7f82bc1d7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 1 monclient: found mon.b 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc1d78c0 con 0x7f82bc13f710 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 10 monclient: _renew_subs 2024-03-25T18:17:07.648 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82bc1d51f0 con 0x7f82bc13f710 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 10 monclient: _renew_subs 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82bc138ad0 con 0x7f82bc13f710 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.646+0000 7f82c460e640 1 librados: init done 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.647+0000 7f82c460e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.647+0000 7f82c460e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.647+0000 7f82c460e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:07.649 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.647+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82bc0627b0 con 0x7f82bc13f710 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.928+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82ac0035f0 con 0x7f82bc13f710 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.928+0000 7f82b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.928+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f82ac003980 con 0x7f82bc13f710 2024-03-25T18:17:07.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.928+0000 7f82b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.928+0000 7f82b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.928+0000 7f82b2ffd640 10 monclient: dump: 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:07.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.928+0000 7f82c0ba7640 4 set_mon_vals no callback set 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.929+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f82ac003ba0 con 0x7f82bc13f710 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.929+0000 7f82c23aa640 10 monclient: get_auth_request con 0x7f82a006e1e0 auth_method 0 2024-03-25T18:17:07.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.930+0000 7f82b2ffd640 10 client.4348.objecter ms_handle_connect 0x7f82a006e1e0 2024-03-25T18:17:07.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.938+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f82ac054790 con 0x7f82bc13f710 2024-03-25T18:17:07.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.938+0000 7f82b2ffd640 10 client.4348.objecter ms_dispatch 0x7f82bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:07.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.938+0000 7f82b2ffd640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:07.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.938+0000 7f82b2ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:07.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.938+0000 7f82b2ffd640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T18:17:07.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.939+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82ac0502a0 con 0x7f82bc13f710 2024-03-25T18:17:07.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.939+0000 7f82b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:07.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:07.939+0000 7f82b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:08.188 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.187+0000 7f82c460e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:08.188 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.187+0000 7f82c460e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:08.188 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.187+0000 7f82c460e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:08.188 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.187+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82bc1d7a50 con 0x7f82bc13f710 2024-03-25T18:17:08.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.209+0000 7f82b2ffd640 1 -- 172.21.15.162:0/4141727356 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f82ac0521d0 con 0x7f82bc13f710 2024-03-25T18:17:08.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.209+0000 7f82b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:08.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.209+0000 7f82b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:08.210 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:17:08.210 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 10 librados: watch_flush enter 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 10 librados: watch_flush exit 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 10 client.4348.objecter successfully canceled tick 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 >> v1:172.21.15.191:6800/1552703795 conn(0x7f82a006e1e0 legacy=0x7f82a00706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 10 monclient: shutdown 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 >> v1:172.21.15.191:6789/0 conn(0x7f82bc13f710 legacy=0x7f82bc1381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 shutdown_connections 2024-03-25T18:17:08.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.211+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 >> 172.21.15.162:0/4141727356 conn(0x7f82bc0b7ea0 msgr2=0x7f82bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:08.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.212+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 shutdown_connections 2024-03-25T18:17:08.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.212+0000 7f82c460e640 1 -- 172.21.15.162:0/4141727356 wait complete. 2024-03-25T18:17:08.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:08.212+0000 7f82c460e640 1 librados: shutdown 2024-03-25T18:17:08.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:15.031 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:15.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.147+0000 7f7b31235640 10 monclient: get_monmap_and_config 2024-03-25T18:17:15.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.147+0000 7f7b31235640 10 monclient: build_initial_monmap 2024-03-25T18:17:15.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.147+0000 7f7b31235640 10 monclient: monmap: 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:17:15.148584+0000 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:17:15.148584+0000 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:17:15.149 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:15.150 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.148+0000 7f7b31235640 1 Processor -- start 2024-03-25T18:17:15.150 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.148+0000 7f7b31235640 1 -- start start 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.148+0000 7f7b31235640 10 monclient: init 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 10 monclient(hunting): picked mon.noname-a con 0x7f7b2c0a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2c13f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 10 monclient(hunting): picked mon.noname-c con 0x7f7b2c1430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c142520 con 0x7f7b2c0a98f0 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0b28a0 con 0x7f7b2c1430c0 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0b2a30 con 0x7f7b2c13f710 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:15.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.149+0000 7f7b31235640 10 monclient(hunting): authenticate will time out at 7734.422941s 2024-03-25T18:17:15.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.159+0000 7f7b2b59e640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f7b2c1430c0 0x7f7b2c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:46466/0 (socket says 172.21.15.162:46466) 2024-03-25T18:17:15.160 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.159+0000 7f7b2b59e640 1 -- 172.21.15.162:0/1030469765 learned_addr learned my addr 172.21.15.162:0/1030469765 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:15.175 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.174+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336583859 0 0) 0x7f7b2c0b2a30 con 0x7f7b2c13f710 2024-03-25T18:17:15.175 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.174+0000 7f7b2a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.175 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.174+0000 7f7b2a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.175 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.174+0000 7f7b2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.175 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.174+0000 7f7b2a59c640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:17:15.175 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.174+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b18003520 con 0x7f7b2c13f710 2024-03-25T18:17:15.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.179+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095806722 0 0) 0x7f7b2c0b28a0 con 0x7f7b2c1430c0 2024-03-25T18:17:15.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.179+0000 7f7b2a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.179+0000 7f7b2a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.179+0000 7f7b2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.179+0000 7f7b2a59c640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:17:15.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.179+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c0b2a30 con 0x7f7b2c1430c0 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2493153477 0 0) 0x7f7b18003520 con 0x7f7b2c13f710 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 >> v1:172.21.15.191:6789/0 conn(0x7f7b2c1430c0 legacy=0x7f7b2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 >> v1:172.21.15.162:6789/0 conn(0x7f7b2c0a98f0 legacy=0x7f7b2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 10 monclient: _finish_hunting 0 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 1 monclient: found mon.noname-b 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0b44c0 con 0x7f7b2c13f710 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 10 monclient: _finish_auth 0 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 10 monclient: _check_auth_tickets 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7b14001690 con 0x7f7b2c13f710 2024-03-25T18:17:15.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b2a59c640 10 monclient: dump: 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b31235640 5 monclient: authenticate success, global_id 4355 2024-03-25T18:17:15.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.189+0000 7f7b31235640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.219+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b140017f0 con 0x7f7b2c13f710 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.219+0000 7f7b2a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.219+0000 7f7b2a59c640 1 -- 172.21.15.162:0/1030469765 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7b14001c00 con 0x7f7b2c13f710 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.219+0000 7f7b2a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.219+0000 7f7b2a59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.219+0000 7f7b2a59c640 10 monclient: dump: 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:15.220 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.219+0000 7f7b31235640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:15.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.219+0000 7f7b31235640 4 set_mon_vals no callback set 2024-03-25T18:17:15.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.220+0000 7f7b31235640 10 monclient: shutdown 2024-03-25T18:17:15.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.220+0000 7f7b31235640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:15.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.220+0000 7f7b31235640 1 -- 172.21.15.162:0/1030469765 >> v1:172.21.15.162:6790/0 conn(0x7f7b2c13f710 legacy=0x7f7b2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.220+0000 7f7b31235640 1 -- 172.21.15.162:0/1030469765 shutdown_connections 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.220+0000 7f7b31235640 1 -- 172.21.15.162:0/1030469765 >> 172.21.15.162:0/1030469765 conn(0x7f7b2c0b7ea0 msgr2=0x7f7b2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.220+0000 7f7b31235640 1 -- 172.21.15.162:0/1030469765 shutdown_connections 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.220+0000 7f7b31235640 1 -- 172.21.15.162:0/1030469765 wait complete. 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.221+0000 7f7b31235640 10 monclient: build_initial_monmap 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.221+0000 7f7b31235640 10 monclient: monmap: 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:17:15.223 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.221+0000 7f7b31235640 1 librados: starting msgr at 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.221+0000 7f7b31235640 1 librados: starting objecter 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.221+0000 7f7b31235640 1 Processor -- start 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 1 -- start start 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 1 librados: setting wanted keys 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 1 librados: calling monclient init 2024-03-25T18:17:15.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 10 monclient: init 2024-03-25T18:17:15.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:15.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:15.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 10 monclient(hunting): picked mon.noname-c con 0x7f7b2c0a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:15.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 10 monclient(hunting): picked mon.noname-a con 0x7f7b2c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:15.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 10 monclient(hunting): picked mon.noname-b con 0x7f7b2c1430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:15.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0b1870 con 0x7f7b2c13f710 2024-03-25T18:17:15.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0ac910 con 0x7f7b2c1430c0 2024-03-25T18:17:15.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b2c0abb00 con 0x7f7b2c0a98f0 2024-03-25T18:17:15.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:15.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.222+0000 7f7b31235640 10 monclient(hunting): authenticate will time out at 7734.496393s 2024-03-25T18:17:15.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.228+0000 7f7b2ad9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f7b2c0a98f0 0x7f7b2c138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:34508/0 (socket says 172.21.15.162:34508) 2024-03-25T18:17:15.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.228+0000 7f7b2ad9d640 1 -- 172.21.15.162:0/1038107838 learned_addr learned my addr 172.21.15.162:0/1038107838 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:15.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.229+0000 7f7b237fe640 10 client.?.objecter ms_handle_connect 0x7f7b2c0a98f0 2024-03-25T18:17:15.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.229+0000 7f7b237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:15.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.231+0000 7f7b237fe640 10 client.?.objecter ms_handle_connect 0x7f7b2c13f710 2024-03-25T18:17:15.232 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.231+0000 7f7b237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:15.233 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.232+0000 7f7b237fe640 10 client.?.objecter ms_handle_connect 0x7f7b2c1430c0 2024-03-25T18:17:15.233 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.232+0000 7f7b237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:15.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.238+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653083122 0 0) 0x7f7b2c0b44c0 con 0x7f7b2c13f710 2024-03-25T18:17:15.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.238+0000 7f7b237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.238+0000 7f7b237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.238+0000 7f7b237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.240 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.238+0000 7f7b237fe640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:17:15.240 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.238+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0c004eb0 con 0x7f7b2c13f710 2024-03-25T18:17:15.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.240+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559204537 0 0) 0x7f7b2c0ac910 con 0x7f7b2c1430c0 2024-03-25T18:17:15.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.240+0000 7f7b237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.240+0000 7f7b237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.240+0000 7f7b237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.240+0000 7f7b237fe640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:17:15.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.240+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c0b44c0 con 0x7f7b2c1430c0 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.250+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685551474 0 0) 0x7f7b2c0abb00 con 0x7f7b2c0a98f0 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.250+0000 7f7b237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.250+0000 7f7b237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.250+0000 7f7b237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.250+0000 7f7b237fe640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.250+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c0ac910 con 0x7f7b2c0a98f0 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.251+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3844001683 0 0) 0x7f7b0c004eb0 con 0x7f7b2c13f710 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.251+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b2c0abb00 con 0x7f7b2c13f710 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.251+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7b14001a20 con 0x7f7b2c13f710 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.251+0000 7f7b237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.251+0000 7f7b237fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.251+0000 7f7b237fe640 10 monclient(hunting): dump: 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:15.252 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:15.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.251+0000 7f7b237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:15.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.258+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036465460 0 0) 0x7f7b2c0b44c0 con 0x7f7b2c1430c0 2024-03-25T18:17:15.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.258+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b0c004eb0 con 0x7f7b2c1430c0 2024-03-25T18:17:15.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.258+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7b24001b90 con 0x7f7b2c1430c0 2024-03-25T18:17:15.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.258+0000 7f7b237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.258+0000 7f7b237fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:15.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.258+0000 7f7b237fe640 10 monclient(hunting): dump: 2024-03-25T18:17:15.259 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:15.259 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:15.260 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.260 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.260 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.260 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:15.260 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:15.260 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:15.260 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:15.260 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:15.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1923758052 0 0) 0x7f7b2c0abb00 con 0x7f7b2c13f710 2024-03-25T18:17:15.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 >> v1:172.21.15.162:6790/0 conn(0x7f7b2c0a98f0 legacy=0x7f7b2c138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 >> v1:172.21.15.191:6789/0 conn(0x7f7b2c1430c0 legacy=0x7f7b2c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:15.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 1 monclient: found mon.a 2024-03-25T18:17:15.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2c0abde0 con 0x7f7b2c13f710 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 10 monclient: _renew_subs 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2c138f40 con 0x7f7b2c13f710 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 10 monclient: _renew_subs 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2c1363f0 con 0x7f7b2c13f710 2024-03-25T18:17:15.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.267+0000 7f7b31235640 1 librados: init done 2024-03-25T18:17:15.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.268+0000 7f7b31235640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:15.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.268+0000 7f7b31235640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:15.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.268+0000 7f7b31235640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:17:15.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.268+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c0627b0 con 0x7f7b2c13f710 2024-03-25T18:17:15.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.274+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b14001c20 con 0x7f7b2c13f710 2024-03-25T18:17:15.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.274+0000 7f7b237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:15.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.274+0000 7f7b29d9b640 4 set_mon_vals no callback set 2024-03-25T18:17:15.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.315+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7b14003a40 con 0x7f7b2c13f710 2024-03-25T18:17:15.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.315+0000 7f7b237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.315+0000 7f7b237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.315+0000 7f7b237fe640 10 monclient: dump: 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:15.316 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:15.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.583+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f7b14019a10 con 0x7f7b2c13f710 2024-03-25T18:17:15.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.584+0000 7f7b2ad9d640 10 monclient: get_auth_request con 0x7f7b0c06df70 auth_method 0 2024-03-25T18:17:15.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.584+0000 7f7b237fe640 10 client.4371.objecter ms_handle_connect 0x7f7b0c06df70 2024-03-25T18:17:15.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.606+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f7b14053620 con 0x7f7b2c13f710 2024-03-25T18:17:15.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.606+0000 7f7b237fe640 10 client.4371.objecter ms_dispatch 0x7f7b2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:15.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.606+0000 7f7b237fe640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:15.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.606+0000 7f7b237fe640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:15.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.606+0000 7f7b237fe640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T18:17:15.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.606+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b1404ef60 con 0x7f7b2c13f710 2024-03-25T18:17:15.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.606+0000 7f7b237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:15.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.606+0000 7f7b237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:15.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.855+0000 7f7b31235640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:15.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.855+0000 7f7b31235640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:15.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.855+0000 7f7b31235640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:17:15.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.855+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b2c1434a0 con 0x7f7b2c13f710 2024-03-25T18:17:15.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.882+0000 7f7b237fe640 1 -- 172.21.15.162:0/1038107838 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f7b14051060 con 0x7f7b2c13f710 2024-03-25T18:17:15.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.882+0000 7f7b237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:15.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.882+0000 7f7b237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:15.883 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:17:15.884 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:15.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.885+0000 7f7b31235640 10 librados: watch_flush enter 2024-03-25T18:17:15.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.885+0000 7f7b31235640 10 librados: watch_flush exit 2024-03-25T18:17:15.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.885+0000 7f7b31235640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T18:17:15.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.885+0000 7f7b31235640 10 client.4371.objecter successfully canceled tick 2024-03-25T18:17:15.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.885+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 >> v1:172.21.15.191:6800/1552703795 conn(0x7f7b0c06df70 legacy=0x7f7b0c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.886+0000 7f7b31235640 10 monclient: shutdown 2024-03-25T18:17:15.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.886+0000 7f7b31235640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:15.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.886+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 >> v1:172.21.15.162:6789/0 conn(0x7f7b2c13f710 legacy=0x7f7b2c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:15.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.886+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 shutdown_connections 2024-03-25T18:17:15.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.886+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 >> 172.21.15.162:0/1038107838 conn(0x7f7b2c0b7ea0 msgr2=0x7f7b2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:15.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.886+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 shutdown_connections 2024-03-25T18:17:15.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.886+0000 7f7b31235640 1 -- 172.21.15.162:0/1038107838 wait complete. 2024-03-25T18:17:15.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:15.886+0000 7f7b31235640 1 librados: shutdown 2024-03-25T18:17:15.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:22.469 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:34.697 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.494 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:34.722 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.529 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:34.757 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.532 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:34.748 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.539 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:34.767 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.545 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:34.761 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.546 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:34.774 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.568 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:34.784 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.571 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:34.799 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:22.705 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:22.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.822+0000 7f44bc2e4640 10 monclient: get_monmap_and_config 2024-03-25T18:17:22.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.822+0000 7f44bc2e4640 10 monclient: build_initial_monmap 2024-03-25T18:17:22.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.822+0000 7f44bc2e4640 10 monclient: monmap: 2024-03-25T18:17:22.823 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:22.823 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:22.824 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:17:22.823664+0000 2024-03-25T18:17:22.824 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:17:22.823664+0000 2024-03-25T18:17:22.824 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:22.824 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:22.824 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:22.824 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:17:22.824 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:17:22.824 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:22.825 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.823+0000 7f44bc2e4640 1 Processor -- start 2024-03-25T18:17:22.825 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.823+0000 7f44bc2e4640 1 -- start start 2024-03-25T18:17:22.825 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.823+0000 7f44bc2e4640 10 monclient: init 2024-03-25T18:17:22.825 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:22.825 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:22.825 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f44b40b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:22.825 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f44b40b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:22.825 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f44b414be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:22.826 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b40af940 con 0x7f44b40b1730 2024-03-25T18:17:22.826 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b40b0a70 con 0x7f44b40b4d60 2024-03-25T18:17:22.826 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b40b0d50 con 0x7f44b414be90 2024-03-25T18:17:22.826 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:22.826 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.824+0000 7f44bc2e4640 10 monclient(hunting): authenticate will time out at 7742.097935s 2024-03-25T18:17:22.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.831+0000 7f44ba080640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f44b40b4d60 0x7f44b413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:51240/0 (socket says 172.21.15.162:51240) 2024-03-25T18:17:22.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.831+0000 7f44ba080640 1 -- 172.21.15.162:0/3688962529 learned_addr learned my addr 172.21.15.162:0/3688962529 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:22.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.846+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418306532 0 0) 0x7f44b40af940 con 0x7f44b40b1730 2024-03-25T18:17:22.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.846+0000 7f44b907e640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.846+0000 7f44b907e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.846+0000 7f44b907e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.846+0000 7f44b907e640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:17:22.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.846+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44a0003520 con 0x7f44b40b1730 2024-03-25T18:17:22.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.851+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880554250 0 0) 0x7f44b40b0a70 con 0x7f44b40b4d60 2024-03-25T18:17:22.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.851+0000 7f44b907e640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.851+0000 7f44b907e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.851+0000 7f44b907e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.851+0000 7f44b907e640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:17:22.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.851+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b40af940 con 0x7f44b40b4d60 2024-03-25T18:17:22.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.860+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381776680 0 0) 0x7f44b40b0d50 con 0x7f44b414be90 2024-03-25T18:17:22.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.861+0000 7f44b907e640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.861+0000 7f44b907e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.861+0000 7f44b907e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.861+0000 7f44b907e640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:17:22.862 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.861+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b40b0a70 con 0x7f44b414be90 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1150408312 0 0) 0x7f44a0003520 con 0x7f44b40b1730 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 >> v1:172.21.15.162:6790/0 conn(0x7f44b414be90 legacy=0x7f44b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 >> v1:172.21.15.191:6789/0 conn(0x7f44b40b4d60 legacy=0x7f44b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 10 monclient: _finish_hunting 0 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 1 monclient: found mon.noname-a 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b40b1030 con 0x7f44b40b1730 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 10 monclient: _finish_auth 0 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 10 monclient: _check_auth_tickets 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:22.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f44a4001690 con 0x7f44b40b1730 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44b907e640 10 monclient: dump: 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44bc2e4640 5 monclient: authenticate success, global_id 4374 2024-03-25T18:17:22.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.864+0000 7f44bc2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44a4001870 con 0x7f44b40b1730 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44b907e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44b907e640 1 -- 172.21.15.162:0/3688962529 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f44a4001c80 con 0x7f44b40b1730 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44b907e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44b907e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44b907e640 10 monclient: dump: 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:22.873 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:22.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44bc2e4640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:22.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44bc2e4640 4 set_mon_vals no callback set 2024-03-25T18:17:22.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44bc2e4640 10 monclient: shutdown 2024-03-25T18:17:22.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44bc2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:22.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44bc2e4640 1 -- 172.21.15.162:0/3688962529 >> v1:172.21.15.162:6789/0 conn(0x7f44b40b1730 legacy=0x7f44b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44bc2e4640 1 -- 172.21.15.162:0/3688962529 shutdown_connections 2024-03-25T18:17:22.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.872+0000 7f44bc2e4640 1 -- 172.21.15.162:0/3688962529 >> 172.21.15.162:0/3688962529 conn(0x7f44b40617a0 msgr2=0x7f44b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:22.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.873+0000 7f44bc2e4640 1 -- 172.21.15.162:0/3688962529 shutdown_connections 2024-03-25T18:17:22.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.873+0000 7f44bc2e4640 1 -- 172.21.15.162:0/3688962529 wait complete. 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.873+0000 7f44bc2e4640 10 monclient: build_initial_monmap 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.873+0000 7f44bc2e4640 10 monclient: monmap: 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 1 librados: starting msgr at 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 1 librados: starting objecter 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 1 Processor -- start 2024-03-25T18:17:22.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 1 -- start start 2024-03-25T18:17:22.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 1 librados: setting wanted keys 2024-03-25T18:17:22.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 1 librados: calling monclient init 2024-03-25T18:17:22.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 10 monclient: init 2024-03-25T18:17:22.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.874+0000 7f44bc2e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.875+0000 7f44bc2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f44b40b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.875+0000 7f44bc2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f44b40b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.875+0000 7f44bc2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f44b414be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.875+0000 7f44bc2e4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b41dadd0 con 0x7f44b40b4d60 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.875+0000 7f44bc2e4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b41398e0 con 0x7f44b40b1730 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.875+0000 7f44bc2e4640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44b413aa60 con 0x7f44b414be90 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.875+0000 7f44bc2e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:22.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.875+0000 7f44bc2e4640 10 monclient(hunting): authenticate will time out at 7742.149066s 2024-03-25T18:17:22.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.878+0000 7f44b987f640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f44b40b4d60 0x7f44b41db2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:34886/0 (socket says 172.21.15.162:34886) 2024-03-25T18:17:22.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.878+0000 7f44b987f640 1 -- 172.21.15.162:0/1075525584 learned_addr learned my addr 172.21.15.162:0/1075525584 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:22.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.879+0000 7f44aaffd640 10 client.?.objecter ms_handle_connect 0x7f44b40b4d60 2024-03-25T18:17:22.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.879+0000 7f44aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:22.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.888+0000 7f44aaffd640 10 client.?.objecter ms_handle_connect 0x7f44b40b1730 2024-03-25T18:17:22.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.888+0000 7f44aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:22.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.890+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515384919 0 0) 0x7f44b41dadd0 con 0x7f44b40b4d60 2024-03-25T18:17:22.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.890+0000 7f44aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.890+0000 7f44aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.890+0000 7f44aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.890+0000 7f44aaffd640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:17:22.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.890+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4498004eb0 con 0x7f44b40b4d60 2024-03-25T18:17:22.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.895+0000 7f44aaffd640 10 client.?.objecter ms_handle_connect 0x7f44b414be90 2024-03-25T18:17:22.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.895+0000 7f44aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:22.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.900+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236723179 0 0) 0x7f44b41398e0 con 0x7f44b40b1730 2024-03-25T18:17:22.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.900+0000 7f44aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.900+0000 7f44aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.900+0000 7f44aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.900+0000 7f44aaffd640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:17:22.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.900+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b41dadd0 con 0x7f44b40b1730 2024-03-25T18:17:22.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.907+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482714407 0 0) 0x7f44b413aa60 con 0x7f44b414be90 2024-03-25T18:17:22.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.907+0000 7f44aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.907+0000 7f44aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.907+0000 7f44aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.907+0000 7f44aaffd640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:17:22.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.907+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44b41398e0 con 0x7f44b414be90 2024-03-25T18:17:22.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.917+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3180392179 0 0) 0x7f44b41398e0 con 0x7f44b414be90 2024-03-25T18:17:22.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.917+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44b413aa60 con 0x7f44b414be90 2024-03-25T18:17:22.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.917+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f44ac001bd0 con 0x7f44b414be90 2024-03-25T18:17:22.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.917+0000 7f44aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.917+0000 7f44aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.917+0000 7f44aaffd640 10 monclient(hunting): dump: 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:22.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.917+0000 7f44aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:22.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.921+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1771367381 0 0) 0x7f44b41dadd0 con 0x7f44b40b1730 2024-03-25T18:17:22.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.921+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44b41398e0 con 0x7f44b40b1730 2024-03-25T18:17:22.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.921+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f44b0001930 con 0x7f44b40b1730 2024-03-25T18:17:22.922 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.921+0000 7f44aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.921+0000 7f44aaffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.921+0000 7f44aaffd640 10 monclient(hunting): dump: 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:22.923 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.922+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 38812568 0 0) 0x7f4498004eb0 con 0x7f44b40b4d60 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.922+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44b41dadd0 con 0x7f44b40b4d60 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.922+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f44a4001c80 con 0x7f44b40b4d60 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.922+0000 7f44aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.922+0000 7f44aaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.922+0000 7f44aaffd640 10 monclient(hunting): dump: 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:22.924 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:22.925 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:22.925 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:22.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1119232546 0 0) 0x7f44b41398e0 con 0x7f44b40b1730 2024-03-25T18:17:22.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 >> v1:172.21.15.162:6790/0 conn(0x7f44b414be90 legacy=0x7f44b413b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 >> v1:172.21.15.162:6789/0 conn(0x7f44b40b4d60 legacy=0x7f44b41db2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:22.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 1 monclient: found mon.b 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b41db9c0 con 0x7f44b40b1730 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 10 monclient: _renew_subs 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b4142d10 con 0x7f44b40b1730 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 10 monclient: _renew_subs 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b413b060 con 0x7f44b40b1730 2024-03-25T18:17:22.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.933+0000 7f44bc2e4640 1 librados: init done 2024-03-25T18:17:22.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.934+0000 7f44bc2e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:22.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.934+0000 7f44bc2e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:22.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.934+0000 7f44bc2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:22.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:22.934+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b41401f0 con 0x7f44b40b1730 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.212+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44b0001d40 con 0x7f44b40b1730 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.212+0000 7f44aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.213+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f44b0004980 con 0x7f44b40b1730 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.213+0000 7f44aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.213+0000 7f44aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.213+0000 7f44aaffd640 10 monclient: dump: 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:23.214 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:23.215 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:23.215 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.215 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:23.215 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:23.215 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:23.215 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:23.215 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:23.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.213+0000 7f44b887d640 4 set_mon_vals no callback set 2024-03-25T18:17:23.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.214+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f44b001aa10 con 0x7f44b40b1730 2024-03-25T18:17:23.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.214+0000 7f44b987f640 10 monclient: get_auth_request con 0x7f449806e1a0 auth_method 0 2024-03-25T18:17:23.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.215+0000 7f44aaffd640 10 client.4360.objecter ms_handle_connect 0x7f449806e1a0 2024-03-25T18:17:23.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.218+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f44b0054730 con 0x7f44b40b1730 2024-03-25T18:17:23.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.218+0000 7f44aaffd640 10 client.4360.objecter ms_dispatch 0x7f44b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:23.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.218+0000 7f44aaffd640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:23.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.218+0000 7f44aaffd640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:23.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.218+0000 7f44aaffd640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T18:17:23.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.219+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44b0050070 con 0x7f44b40b1730 2024-03-25T18:17:23.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.219+0000 7f44aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:23.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.219+0000 7f44aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:23.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.460+0000 7f44bc2e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:23.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.460+0000 7f44bc2e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:23.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.460+0000 7f44bc2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:23.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.460+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44b41dac30 con 0x7f44b40b1730 2024-03-25T18:17:23.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.654+0000 7f44aaffd640 1 -- 172.21.15.162:0/1075525584 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f44b41dac30 con 0x7f44b40b1730 2024-03-25T18:17:23.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.654+0000 7f44aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:23.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.654+0000 7f44aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:23.655 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:17:23.655 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 10 librados: watch_flush enter 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 10 librados: watch_flush exit 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 10 client.4360.objecter successfully canceled tick 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 >> v1:172.21.15.191:6800/1552703795 conn(0x7f449806e1a0 legacy=0x7f4498070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 10 monclient: shutdown 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 >> v1:172.21.15.191:6789/0 conn(0x7f44b40b1730 legacy=0x7f44b41391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 shutdown_connections 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 >> 172.21.15.162:0/1075525584 conn(0x7f44b40617a0 msgr2=0x7f44b4137490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 shutdown_connections 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 1 -- 172.21.15.162:0/1075525584 wait complete. 2024-03-25T18:17:23.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:23.656+0000 7f44bc2e4640 1 librados: shutdown 2024-03-25T18:17:23.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:30.475 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.580+0000 7fa1f2a4a640 10 monclient: get_monmap_and_config 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.580+0000 7fa1f2a4a640 10 monclient: build_initial_monmap 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.580+0000 7fa1f2a4a640 10 monclient: monmap: 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:17:30.581319+0000 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:17:30.581319+0000 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:30.581 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:17:30.582 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:17:30.582 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:30.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.580+0000 7fa1f2a4a640 1 Processor -- start 2024-03-25T18:17:30.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 1 -- start start 2024-03-25T18:17:30.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 10 monclient: init 2024-03-25T18:17:30.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:30.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:30.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1ec070410 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:30.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fa1ec0707f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:30.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fa1ec13db40 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:30.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0adea0 con 0x7fa1ec13db40 2024-03-25T18:17:30.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec1445a0 con 0x7fa1ec070410 2024-03-25T18:17:30.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0720a0 con 0x7fa1ec0707f0 2024-03-25T18:17:30.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:30.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.581+0000 7fa1f2a4a640 10 monclient(hunting): authenticate will time out at 7749.855549s 2024-03-25T18:17:30.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.582+0000 7fa1eb7fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fa1ec0707f0 0x7fa1ec13d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:33438/0 (socket says 172.21.15.162:33438) 2024-03-25T18:17:30.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.582+0000 7fa1eb7fe640 1 -- 172.21.15.162:0/3663563308 learned_addr learned my addr 172.21.15.162:0/3663563308 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:30.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.633+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353312744 0 0) 0x7fa1ec0adea0 con 0x7fa1ec13db40 2024-03-25T18:17:30.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.633+0000 7fa1eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:30.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.633+0000 7fa1eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:30.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.633+0000 7fa1eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:30.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.633+0000 7fa1eaffd640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:17:30.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.633+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d4003520 con 0x7fa1ec13db40 2024-03-25T18:17:30.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2373015704 0 0) 0x7fa1d4003520 con 0x7fa1ec13db40 2024-03-25T18:17:30.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 >> v1:172.21.15.162:6790/0 conn(0x7fa1ec0707f0 legacy=0x7fa1ec13d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 >> v1:172.21.15.191:6789/0 conn(0x7fa1ec070410 legacy=0x7fa1ec143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 1 monclient: found mon.noname-a 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec072380 con 0x7fa1ec13db40 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa1e4001a10 con 0x7fa1ec13db40 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1f2a4a640 5 monclient: authenticate success, global_id 4380 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1f2a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1eaffd640 10 monclient: dump: 2024-03-25T18:17:30.644 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:30.645 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.642+0000 7fa1f2a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.657+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1e4001b70 con 0x7fa1ec13db40 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.657+0000 7fa1eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.657+0000 7fa1eaffd640 1 -- 172.21.15.162:0/3663563308 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa1e4001f80 con 0x7fa1ec13db40 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.657+0000 7fa1f2a4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.657+0000 7fa1f2a4a640 4 set_mon_vals no callback set 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.657+0000 7fa1eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.657+0000 7fa1eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.657+0000 7fa1eaffd640 10 monclient: dump: 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:30.658 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.659 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.659 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.659 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:30.659 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:30.659 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:30.659 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:30.659 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:30.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.658+0000 7fa1f2a4a640 10 monclient: shutdown 2024-03-25T18:17:30.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.658+0000 7fa1f2a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.658+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3663563308 >> v1:172.21.15.162:6789/0 conn(0x7fa1ec13db40 legacy=0x7fa1ec14ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.658+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3663563308 shutdown_connections 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.658+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3663563308 >> 172.21.15.162:0/3663563308 conn(0x7fa1ec06b630 msgr2=0x7fa1ec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.658+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3663563308 shutdown_connections 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.658+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3663563308 wait complete. 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.659+0000 7fa1f2a4a640 10 monclient: build_initial_monmap 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.659+0000 7fa1f2a4a640 10 monclient: monmap: 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:30.661 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.659+0000 7fa1f2a4a640 1 librados: starting msgr at 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.659+0000 7fa1f2a4a640 1 librados: starting objecter 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.659+0000 7fa1f2a4a640 1 Processor -- start 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 1 -- start start 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 1 librados: setting wanted keys 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 1 librados: calling monclient init 2024-03-25T18:17:30.662 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 10 monclient: init 2024-03-25T18:17:30.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fa1ec070410 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fa1ec0707f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fa1ec13db40 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec1d52d0 con 0x7fa1ec0707f0 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec136610 con 0x7fa1ec070410 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec13dfc0 con 0x7fa1ec13db40 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:30.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.660+0000 7fa1f2a4a640 10 monclient(hunting): authenticate will time out at 7749.934631s 2024-03-25T18:17:30.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.666+0000 7fa1eb7fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa1ec0707f0 0x7fa1ec1357f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:49228/0 (socket says 172.21.15.162:49228) 2024-03-25T18:17:30.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.666+0000 7fa1eb7fe640 1 -- 172.21.15.162:0/3668426055 learned_addr learned my addr 172.21.15.162:0/3668426055 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:30.670 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.669+0000 7fa1e8ff9640 10 client.?.objecter ms_handle_connect 0x7fa1ec0707f0 2024-03-25T18:17:30.670 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.669+0000 7fa1e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:30.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.677+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480388148 0 0) 0x7fa1ec1d52d0 con 0x7fa1ec0707f0 2024-03-25T18:17:30.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.677+0000 7fa1e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:30.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.677+0000 7fa1e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:30.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.677+0000 7fa1e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:30.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.677+0000 7fa1e8ff9640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:17:30.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.677+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8004eb0 con 0x7fa1ec0707f0 2024-03-25T18:17:30.681 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.680+0000 7fa1e8ff9640 10 client.?.objecter ms_handle_connect 0x7fa1ec070410 2024-03-25T18:17:30.681 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.680+0000 7fa1e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:30.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.690+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158755586 0 0) 0x7fa1ec136610 con 0x7fa1ec070410 2024-03-25T18:17:30.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.690+0000 7fa1e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:30.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.690+0000 7fa1e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:30.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.690+0000 7fa1e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:30.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.690+0000 7fa1e8ff9640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:17:30.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.690+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ec1d52d0 con 0x7fa1ec070410 2024-03-25T18:17:30.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 10 client.?.objecter ms_handle_connect 0x7fa1ec13db40 2024-03-25T18:17:30.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:30.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3849511714 0 0) 0x7fa1d8004eb0 con 0x7fa1ec0707f0 2024-03-25T18:17:30.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1ec136610 con 0x7fa1ec0707f0 2024-03-25T18:17:30.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa1e0001930 con 0x7fa1ec0707f0 2024-03-25T18:17:30.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:30.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:30.695 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.693+0000 7fa1e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:30.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.699+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90556690 0 0) 0x7fa1ec13dfc0 con 0x7fa1ec13db40 2024-03-25T18:17:30.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.699+0000 7fa1e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:30.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.699+0000 7fa1e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:30.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.699+0000 7fa1e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:30.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.699+0000 7fa1e8ff9640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:17:30.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.699+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d8004eb0 con 0x7fa1ec13db40 2024-03-25T18:17:30.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.708+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2729892602 0 0) 0x7fa1ec1d52d0 con 0x7fa1ec070410 2024-03-25T18:17:30.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.708+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1ec13dfc0 con 0x7fa1ec070410 2024-03-25T18:17:30.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.708+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa1dc0018a0 con 0x7fa1ec070410 2024-03-25T18:17:30.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.708+0000 7fa1e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.708+0000 7fa1e8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:30.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.708+0000 7fa1e8ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:30.709 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:30.709 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:30.710 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.710 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.710 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.710 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:30.710 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:30.710 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:30.710 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:30.710 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.714+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3085018450 0 0) 0x7fa1d8004eb0 con 0x7fa1ec13db40 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.714+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1ec1d52d0 con 0x7fa1ec13db40 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.714+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa1e4001e80 con 0x7fa1ec13db40 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.714+0000 7fa1e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.714+0000 7fa1e8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.714+0000 7fa1e8ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.715 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.716 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:30.716 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:30.716 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:30.716 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:30.716 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1920355074 0 0) 0x7fa1ec13dfc0 con 0x7fa1ec070410 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 >> v1:172.21.15.162:6790/0 conn(0x7fa1ec13db40 legacy=0x7fa1ec135f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 >> v1:172.21.15.162:6789/0 conn(0x7fa1ec0707f0 legacy=0x7fa1ec1357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 1 monclient: found mon.b 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec140f70 con 0x7fa1ec070410 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:30.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 10 monclient: _renew_subs 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec0ab630 con 0x7fa1ec070410 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 10 monclient: _renew_subs 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec1d6050 con 0x7fa1ec070410 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.719+0000 7fa1f2a4a640 1 librados: init done 2024-03-25T18:17:30.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.720+0000 7fa1f2a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:30.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.720+0000 7fa1f2a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:30.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.720+0000 7fa1f2a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:30.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.720+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec0627b0 con 0x7fa1ec070410 2024-03-25T18:17:30.742 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.741+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1dc0035f0 con 0x7fa1ec070410 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.741+0000 7fa1e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.741+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa1dc003940 con 0x7fa1ec070410 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.741+0000 7fa1e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.741+0000 7fa1e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.741+0000 7fa1e8ff9640 10 monclient: dump: 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.743 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:30.744 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:30.744 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:30.744 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:30.744 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:30.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.741+0000 7fa1ea7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:30.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.742+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fa1dc003b60 con 0x7fa1ec070410 2024-03-25T18:17:30.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.743+0000 7fa1eb7fe640 10 monclient: get_auth_request con 0x7fa1d806e110 auth_method 0 2024-03-25T18:17:30.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.743+0000 7fa1e8ff9640 10 client.4366.objecter ms_handle_connect 0x7fa1d806e110 2024-03-25T18:17:30.745 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.744+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fa1dc0534d0 con 0x7fa1ec070410 2024-03-25T18:17:30.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.744+0000 7fa1e8ff9640 10 client.4366.objecter ms_dispatch 0x7fa1ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:30.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.744+0000 7fa1e8ff9640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:30.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.744+0000 7fa1e8ff9640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:30.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:30.745+0000 7fa1e8ff9640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T18:17:31.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.006+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1dc04efe0 con 0x7fa1ec070410 2024-03-25T18:17:31.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.006+0000 7fa1e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:31.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.006+0000 7fa1e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:31.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.253+0000 7fa1f2a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:31.255 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.253+0000 7fa1f2a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:31.255 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.253+0000 7fa1f2a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:31.255 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.253+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1ec0681e0 con 0x7fa1ec070410 2024-03-25T18:17:31.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.282+0000 7fa1e8ff9640 1 -- 172.21.15.162:0/3668426055 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fa1dc050f10 con 0x7fa1ec070410 2024-03-25T18:17:31.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.282+0000 7fa1e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:31.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.282+0000 7fa1e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:31.283 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:17:31.283 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 10 librados: watch_flush enter 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 10 librados: watch_flush exit 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 10 client.4366.objecter successfully canceled tick 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 >> v1:172.21.15.191:6800/1552703795 conn(0x7fa1d806e110 legacy=0x7fa1d80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 10 monclient: shutdown 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 >> v1:172.21.15.191:6789/0 conn(0x7fa1ec070410 legacy=0x7fa1ec1d5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 shutdown_connections 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 >> 172.21.15.162:0/3668426055 conn(0x7fa1ec06b630 msgr2=0x7fa1ec070f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:31.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.284+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 shutdown_connections 2024-03-25T18:17:31.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.285+0000 7fa1f2a4a640 1 -- 172.21.15.162:0/3668426055 wait complete. 2024-03-25T18:17:31.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:31.285+0000 7fa1f2a4a640 1 librados: shutdown 2024-03-25T18:17:31.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:38.103 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:38.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.216+0000 7f8f0e5a5640 10 monclient: get_monmap_and_config 2024-03-25T18:17:38.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.216+0000 7f8f0e5a5640 10 monclient: build_initial_monmap 2024-03-25T18:17:38.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.216+0000 7f8f0e5a5640 10 monclient: monmap: 2024-03-25T18:17:38.217 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:38.217 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:38.217 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:17:38.217473+0000 2024-03-25T18:17:38.217 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:17:38.217473+0000 2024-03-25T18:17:38.218 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:38.218 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:38.218 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:38.218 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:17:38.218 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:17:38.218 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:38.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.217+0000 7f8f0e5a5640 1 Processor -- start 2024-03-25T18:17:38.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.217+0000 7f8f0e5a5640 1 -- start start 2024-03-25T18:17:38.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.217+0000 7f8f0e5a5640 10 monclient: init 2024-03-25T18:17:38.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.217+0000 7f8f0e5a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:38.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.217+0000 7f8f0e5a5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:38.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.218+0000 7f8f0e5a5640 10 monclient(hunting): picked mon.noname-b con 0x7f8f080a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:38.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.218+0000 7f8f0e5a5640 10 monclient(hunting): picked mon.noname-a con 0x7f8f0813f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:38.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.218+0000 7f8f0e5a5640 10 monclient(hunting): picked mon.noname-c con 0x7f8f081430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:38.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.218+0000 7f8f0e5a5640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f08142520 con 0x7f8f0813f710 2024-03-25T18:17:38.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.218+0000 7f8f0e5a5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f080b28a0 con 0x7f8f081430c0 2024-03-25T18:17:38.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.218+0000 7f8f0e5a5640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f080b2a30 con 0x7f8f080a98f0 2024-03-25T18:17:38.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.218+0000 7f8f0e5a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:38.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.218+0000 7f8f0e5a5640 10 monclient(hunting): authenticate will time out at 7757.491885s 2024-03-25T18:17:38.231 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.230+0000 7f8f07fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f8f080a98f0 0x7f8f080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:35110/0 (socket says 172.21.15.162:35110) 2024-03-25T18:17:38.231 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.230+0000 7f8f07fff640 1 -- 172.21.15.162:0/4086399139 learned_addr learned my addr 172.21.15.162:0/4086399139 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:38.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.247+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542357345 0 0) 0x7f8f08142520 con 0x7f8f0813f710 2024-03-25T18:17:38.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.247+0000 7f8f06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.247+0000 7f8f06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.247+0000 7f8f06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.247+0000 7f8f06ffd640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:17:38.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.247+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef0003520 con 0x7f8f0813f710 2024-03-25T18:17:38.249 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77248952 0 0) 0x7f8f080b2a30 con 0x7f8f080a98f0 2024-03-25T18:17:38.249 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.249 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.249 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.249 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:17:38.249 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f08142520 con 0x7f8f080a98f0 2024-03-25T18:17:38.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499820848 0 0) 0x7f8f080b28a0 con 0x7f8f081430c0 2024-03-25T18:17:38.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:17:38.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.248+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f080b2a30 con 0x7f8f081430c0 2024-03-25T18:17:38.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.258+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007528016 0 0) 0x7f8ef0003520 con 0x7f8f0813f710 2024-03-25T18:17:38.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.258+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 >> v1:172.21.15.162:6790/0 conn(0x7f8f080a98f0 legacy=0x7f8f080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.258+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 >> v1:172.21.15.191:6789/0 conn(0x7f8f081430c0 legacy=0x7f8f0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.258+0000 7f8f06ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:38.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.258+0000 7f8f06ffd640 1 monclient: found mon.noname-a 2024-03-25T18:17:38.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:38.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:17:38.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f080b44c0 con 0x7f8f0813f710 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8efc001690 con 0x7f8f0813f710 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f06ffd640 10 monclient: dump: 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:38.261 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.262 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:38.262 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:38.262 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:38.262 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:38.262 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:38.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f0e5a5640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:17:38.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.259+0000 7f8f0e5a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:38.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.274+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8efc0017f0 con 0x7f8f0813f710 2024-03-25T18:17:38.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.274+0000 7f8f06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:38.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.274+0000 7f8f06ffd640 1 -- 172.21.15.162:0/4086399139 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8efc001c00 con 0x7f8f0813f710 2024-03-25T18:17:38.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.274+0000 7f8f0e5a5640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.274+0000 7f8f0e5a5640 4 set_mon_vals no callback set 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.274+0000 7f8f06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.274+0000 7f8f06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.274+0000 7f8f06ffd640 10 monclient: dump: 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:38.276 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:38.277 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:38.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.275+0000 7f8f0e5a5640 10 monclient: shutdown 2024-03-25T18:17:38.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.275+0000 7f8f0e5a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:38.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.275+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/4086399139 >> v1:172.21.15.162:6789/0 conn(0x7f8f0813f710 legacy=0x7f8f081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.275+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/4086399139 shutdown_connections 2024-03-25T18:17:38.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.275+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/4086399139 >> 172.21.15.162:0/4086399139 conn(0x7f8f080b7ea0 msgr2=0x7f8f080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:38.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.275+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/4086399139 shutdown_connections 2024-03-25T18:17:38.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.275+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/4086399139 wait complete. 2024-03-25T18:17:38.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 10 monclient: build_initial_monmap 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 10 monclient: monmap: 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:17:38.279 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:38.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 1 librados: starting msgr at 2024-03-25T18:17:38.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 1 librados: starting objecter 2024-03-25T18:17:38.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 1 Processor -- start 2024-03-25T18:17:38.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 1 -- start start 2024-03-25T18:17:38.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 1 librados: setting wanted keys 2024-03-25T18:17:38.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 1 librados: calling monclient init 2024-03-25T18:17:38.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.276+0000 7f8f0e5a5640 10 monclient: init 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 10 monclient(hunting): picked mon.noname-b con 0x7f8f080a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 10 monclient(hunting): picked mon.noname-c con 0x7f8f0813f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 10 monclient(hunting): picked mon.noname-a con 0x7f8f081430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f08135550 con 0x7f8f081430c0 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f08136b50 con 0x7f8f080a98f0 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f08136e30 con 0x7f8f0813f710 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:38.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.277+0000 7f8f0e5a5640 10 monclient(hunting): authenticate will time out at 7757.551562s 2024-03-25T18:17:38.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.282+0000 7f8f077fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f8f0813f710 0x7f8f081375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:35138/0 (socket says 172.21.15.162:35138) 2024-03-25T18:17:38.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.282+0000 7f8f077fe640 1 -- 172.21.15.162:0/2044224514 learned_addr learned my addr 172.21.15.162:0/2044224514 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:38.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.287+0000 7f8f04ff9640 10 client.?.objecter ms_handle_connect 0x7f8f0813f710 2024-03-25T18:17:38.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.287+0000 7f8f04ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:38.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.293+0000 7f8f04ff9640 10 client.?.objecter ms_handle_connect 0x7f8f081430c0 2024-03-25T18:17:38.294 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.293+0000 7f8f04ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:38.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.297+0000 7f8f04ff9640 10 client.?.objecter ms_handle_connect 0x7f8f080a98f0 2024-03-25T18:17:38.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.297+0000 7f8f04ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:38.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076365860 0 0) 0x7f8f08136b50 con 0x7f8f080a98f0 2024-03-25T18:17:38.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:17:38.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef4004eb0 con 0x7f8f080a98f0 2024-03-25T18:17:38.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102884701 0 0) 0x7f8f08136e30 con 0x7f8f0813f710 2024-03-25T18:17:38.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:17:38.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.303+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f08136b50 con 0x7f8f0813f710 2024-03-25T18:17:38.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.307+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569787267 0 0) 0x7f8f08135550 con 0x7f8f081430c0 2024-03-25T18:17:38.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.307+0000 7f8f04ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:38.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.307+0000 7f8f04ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:38.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.307+0000 7f8f04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:38.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.307+0000 7f8f04ff9640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:17:38.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.307+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f08136e30 con 0x7f8f081430c0 2024-03-25T18:17:38.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.316+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 363119027 0 0) 0x7f8ef4004eb0 con 0x7f8f080a98f0 2024-03-25T18:17:38.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.316+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f08135550 con 0x7f8f080a98f0 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.316+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8ef8001970 con 0x7f8f080a98f0 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.316+0000 7f8f04ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.316+0000 7f8f04ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.316+0000 7f8f04ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:38.318 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:38.319 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:38.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.316+0000 7f8f04ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:38.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.318+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2074351787 0 0) 0x7f8f08136b50 con 0x7f8f0813f710 2024-03-25T18:17:38.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.318+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ef4004eb0 con 0x7f8f0813f710 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.320+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 872897969 0 0) 0x7f8f08135550 con 0x7f8f080a98f0 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 >> v1:172.21.15.162:6790/0 conn(0x7f8f0813f710 legacy=0x7f8f081375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 >> v1:172.21.15.162:6789/0 conn(0x7f8f081430c0 legacy=0x7f8f081e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 1 monclient: found mon.b 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f081357f0 con 0x7f8f080a98f0 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:38.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 10 monclient: _renew_subs 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f08137cb0 con 0x7f8f080a98f0 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 10 monclient: _renew_subs 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f081352e0 con 0x7f8f080a98f0 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.321+0000 7f8f0e5a5640 1 librados: init done 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.322+0000 7f8f0e5a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.322+0000 7f8f0e5a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.322+0000 7f8f0e5a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:38.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.322+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f080627b0 con 0x7f8f080a98f0 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.588+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ef80047e0 con 0x7f8f080a98f0 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.588+0000 7f8f04ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.588+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8ef8004af0 con 0x7f8f080a98f0 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.588+0000 7f8f04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.588+0000 7f8f04ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.588+0000 7f8f067fc640 4 set_mon_vals no callback set 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.588+0000 7f8f04ff9640 10 monclient: dump: 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:38.589 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:38.590 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:38.590 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:38.590 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:38.590 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:38.590 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:38.590 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:38.590 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:38.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.588+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f8ef801aac0 con 0x7f8f080a98f0 2024-03-25T18:17:38.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.589+0000 7f8f077fe640 10 monclient: get_auth_request con 0x7f8ef406e070 auth_method 0 2024-03-25T18:17:38.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.589+0000 7f8f04ff9640 10 client.4372.objecter ms_handle_connect 0x7f8ef406e070 2024-03-25T18:17:38.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.594+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f8ef80547a0 con 0x7f8f080a98f0 2024-03-25T18:17:38.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.594+0000 7f8f04ff9640 10 client.4372.objecter ms_dispatch 0x7f8f08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:38.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.594+0000 7f8f04ff9640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:38.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.594+0000 7f8f04ff9640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:38.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.594+0000 7f8f04ff9640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T18:17:38.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.596+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ef80500e0 con 0x7f8f080a98f0 2024-03-25T18:17:38.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.596+0000 7f8f04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:38.611 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.596+0000 7f8f04ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:38.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.809+0000 7f8f0e5a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:38.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.809+0000 7f8f0e5a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:38.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.809+0000 7f8f0e5a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:38.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.809+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f08135980 con 0x7f8f080a98f0 2024-03-25T18:17:38.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.859+0000 7f8f04ff9640 1 -- 172.21.15.162:0/2044224514 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f8ef80521e0 con 0x7f8f080a98f0 2024-03-25T18:17:38.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.859+0000 7f8f04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:38.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.859+0000 7f8f04ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:38.861 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:17:38.861 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:38.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 10 librados: watch_flush enter 2024-03-25T18:17:38.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 10 librados: watch_flush exit 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 10 client.4372.objecter successfully canceled tick 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 >> v1:172.21.15.191:6800/1552703795 conn(0x7f8ef406e070 legacy=0x7f8ef4070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 10 monclient: shutdown 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 >> v1:172.21.15.191:6789/0 conn(0x7f8f080a98f0 legacy=0x7f8f0813b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 shutdown_connections 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 >> 172.21.15.162:0/2044224514 conn(0x7f8f080b7ea0 msgr2=0x7f8f08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 shutdown_connections 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.862+0000 7f8f0e5a5640 1 -- 172.21.15.162:0/2044224514 wait complete. 2024-03-25T18:17:38.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:38.863+0000 7f8f0e5a5640 1 librados: shutdown 2024-03-25T18:17:38.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:42.558 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:54.785 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.565 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:54.792 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.565 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:54.793 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.605 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:03:54.833 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.622 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:54.839 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.627 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:54.855 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.630 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:54.847 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.658 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:03:54.874 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:42.660 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:03:54.888 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.682 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:45.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.796+0000 7f3cf9d6b640 10 monclient: get_monmap_and_config 2024-03-25T18:17:45.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.796+0000 7f3cf9d6b640 10 monclient: build_initial_monmap 2024-03-25T18:17:45.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.796+0000 7f3cf9d6b640 10 monclient: monmap: 2024-03-25T18:17:45.797 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:45.797 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:45.797 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:17:45.797337+0000 2024-03-25T18:17:45.797 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:17:45.797337+0000 2024-03-25T18:17:45.797 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:45.798 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:45.798 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:45.798 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:17:45.798 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:17:45.798 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 1 Processor -- start 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 1 -- start start 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 10 monclient: init 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf40b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf40b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf414be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf40af940 con 0x7f3cf414be90 2024-03-25T18:17:45.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf40b0a70 con 0x7f3cf40b1730 2024-03-25T18:17:45.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf40b0d50 con 0x7f3cf40b4d60 2024-03-25T18:17:45.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:45.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.797+0000 7f3cf9d6b640 10 monclient(hunting): authenticate will time out at 7765.071641s 2024-03-25T18:17:45.806 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.805+0000 7f3cf3fff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f3cf414be90 0x7f3cf414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:36172/0 (socket says 172.21.15.162:36172) 2024-03-25T18:17:45.806 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.805+0000 7f3cf3fff640 1 -- 172.21.15.162:0/4159389408 learned_addr learned my addr 172.21.15.162:0/4159389408 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:45.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.822+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901157311 0 0) 0x7f3cf40af940 con 0x7f3cf414be90 2024-03-25T18:17:45.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.822+0000 7f3cf27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:45.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.822+0000 7f3cf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:45.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.822+0000 7f3cf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:45.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.822+0000 7f3cf27fc640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:17:45.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.822+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc003520 con 0x7f3cf414be90 2024-03-25T18:17:45.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 391101587 0 0) 0x7f3cdc003520 con 0x7f3cf414be90 2024-03-25T18:17:45.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 >> v1:172.21.15.162:6790/0 conn(0x7f3cf40b4d60 legacy=0x7f3cf413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 >> v1:172.21.15.191:6789/0 conn(0x7f3cf40b1730 legacy=0x7f3cf40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:45.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 1 monclient: found mon.noname-a 2024-03-25T18:17:45.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:45.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf40b1030 con 0x7f3cf414be90 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3cec001a10 con 0x7f3cf414be90 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf27fc640 10 monclient: dump: 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:45.836 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:45.837 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:45.837 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:45.837 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:45.837 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:45.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf9d6b640 5 monclient: authenticate success, global_id 4392 2024-03-25T18:17:45.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.834+0000 7f3cf9d6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cec001b70 con 0x7f3cf414be90 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf27fc640 1 -- 172.21.15.162:0/4159389408 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3cec001f80 con 0x7f3cf414be90 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 4 set_mon_vals no callback set 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf27fc640 10 monclient: dump: 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:45.853 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:45.854 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:45.854 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:45.854 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:45.854 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:45.854 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:45.854 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 10 monclient: shutdown 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/4159389408 >> v1:172.21.15.162:6789/0 conn(0x7f3cf414be90 legacy=0x7f3cf414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/4159389408 shutdown_connections 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/4159389408 >> 172.21.15.162:0/4159389408 conn(0x7f3cf40617a0 msgr2=0x7f3cf4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/4159389408 shutdown_connections 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.852+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/4159389408 wait complete. 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.853+0000 7f3cf9d6b640 10 monclient: build_initial_monmap 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.853+0000 7f3cf9d6b640 10 monclient: monmap: 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:45.856 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.854+0000 7f3cf9d6b640 1 librados: starting msgr at 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.854+0000 7f3cf9d6b640 1 librados: starting objecter 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.854+0000 7f3cf9d6b640 1 Processor -- start 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.854+0000 7f3cf9d6b640 1 -- start start 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.854+0000 7f3cf9d6b640 1 librados: setting wanted keys 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.854+0000 7f3cf9d6b640 1 librados: calling monclient init 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.854+0000 7f3cf9d6b640 10 monclient: init 2024-03-25T18:17:45.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf40b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf40b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf414be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf4140810 con 0x7f3cf414be90 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf4140380 con 0x7f3cf40b1730 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf4140510 con 0x7f3cf40b4d60 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:45.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.855+0000 7f3cf9d6b640 10 monclient(hunting): authenticate will time out at 7765.128766s 2024-03-25T18:17:45.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.858+0000 7f3cf37fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3cf40b1730 0x7f3cf4134ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:60530/0 (socket says 172.21.15.162:60530) 2024-03-25T18:17:45.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.858+0000 7f3cf37fe640 1 -- 172.21.15.162:0/3220436391 learned_addr learned my addr 172.21.15.162:0/3220436391 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:45.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.860+0000 7f3cdbfff640 10 client.?.objecter ms_handle_connect 0x7f3cf414be90 2024-03-25T18:17:45.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.860+0000 7f3cdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:45.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.860+0000 7f3cdbfff640 10 client.?.objecter ms_handle_connect 0x7f3cf40b1730 2024-03-25T18:17:45.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.860+0000 7f3cdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:45.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.866+0000 7f3cdbfff640 10 client.?.objecter ms_handle_connect 0x7f3cf40b4d60 2024-03-25T18:17:45.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.866+0000 7f3cdbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:45.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.870+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735385980 0 0) 0x7f3cf4140380 con 0x7f3cf40b1730 2024-03-25T18:17:45.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.870+0000 7f3cdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:45.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.870+0000 7f3cdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:45.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.870+0000 7f3cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:45.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.870+0000 7f3cdbfff640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:17:45.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.870+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce0004eb0 con 0x7f3cf40b1730 2024-03-25T18:17:45.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.876+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385464869 0 0) 0x7f3cf4140810 con 0x7f3cf414be90 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.876+0000 7f3cdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.876+0000 7f3cdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.876+0000 7f3cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.876+0000 7f3cdbfff640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.876+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf4140380 con 0x7f3cf414be90 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.877+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462888 0 0) 0x7f3cf4140510 con 0x7f3cf40b4d60 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.877+0000 7f3cdbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.877+0000 7f3cdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.877+0000 7f3cdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.877+0000 7f3cdbfff640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:17:45.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.877+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cf4140810 con 0x7f3cf40b4d60 2024-03-25T18:17:45.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.879+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3321355962 0 0) 0x7f3ce0004eb0 con 0x7f3cf40b1730 2024-03-25T18:17:45.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.879+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cf4140510 con 0x7f3cf40b1730 2024-03-25T18:17:45.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.879+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3ce40019d0 con 0x7f3cf40b1730 2024-03-25T18:17:45.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.879+0000 7f3cdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:45.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.879+0000 7f3cdbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:45.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.879+0000 7f3cdbfff640 10 monclient(hunting): dump: 2024-03-25T18:17:45.880 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:45.880 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:45.881 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.879+0000 7f3cdbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:45.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.889+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1596905877 0 0) 0x7f3cf4140810 con 0x7f3cf40b4d60 2024-03-25T18:17:45.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.889+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ce0004eb0 con 0x7f3cf40b4d60 2024-03-25T18:17:45.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2700437773 0 0) 0x7f3cf4140510 con 0x7f3cf40b1730 2024-03-25T18:17:45.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 >> v1:172.21.15.162:6790/0 conn(0x7f3cf40b4d60 legacy=0x7f3cf41d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 >> v1:172.21.15.162:6789/0 conn(0x7f3cf414be90 legacy=0x7f3cf41e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:45.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 1 monclient: found mon.b 2024-03-25T18:17:45.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf41355e0 con 0x7f3cf40b1730 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cf9d6b640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cf9d6b640 10 monclient: _renew_subs 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cf9d6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.890+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf41357d0 con 0x7f3cf40b1730 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.891+0000 7f3cf9d6b640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.891+0000 7f3cf9d6b640 10 monclient: _renew_subs 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.891+0000 7f3cf9d6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.891+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf4144da0 con 0x7f3cf40b1730 2024-03-25T18:17:45.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.891+0000 7f3cf9d6b640 1 librados: init done 2024-03-25T18:17:45.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.892+0000 7f3cf9d6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:45.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.892+0000 7f3cf9d6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:45.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.892+0000 7f3cf9d6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:45.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:45.892+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf4141c10 con 0x7f3cf40b1730 2024-03-25T18:17:46.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.111+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ce4003610 con 0x7f3cf40b1730 2024-03-25T18:17:46.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.111+0000 7f3cdbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:46.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.111+0000 7f3cf1ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:46.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.167+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3ce4003a20 con 0x7f3cf40b1730 2024-03-25T18:17:46.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.167+0000 7f3cdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:46.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.167+0000 7f3cdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:46.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.167+0000 7f3cdbfff640 10 monclient: dump: 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:46.169 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:46.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.168+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f3ce4003c00 con 0x7f3cf40b1730 2024-03-25T18:17:46.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.169+0000 7f3cf2ffd640 10 monclient: get_auth_request con 0x7f3ce006e120 auth_method 0 2024-03-25T18:17:46.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.169+0000 7f3cdbfff640 10 client.4378.objecter ms_handle_connect 0x7f3ce006e120 2024-03-25T18:17:46.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.177+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f3ce4054880 con 0x7f3cf40b1730 2024-03-25T18:17:46.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.177+0000 7f3cdbfff640 10 client.4378.objecter ms_dispatch 0x7f3cf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:46.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.177+0000 7f3cdbfff640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:46.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.177+0000 7f3cdbfff640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:46.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.177+0000 7f3cdbfff640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T18:17:46.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.180+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ce4050390 con 0x7f3cf40b1730 2024-03-25T18:17:46.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.180+0000 7f3cdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:46.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.180+0000 7f3cdbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:46.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.434+0000 7f3cf9d6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:46.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.434+0000 7f3cf9d6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:46.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.434+0000 7f3cf9d6b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:17:46.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.434+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cf40627b0 con 0x7f3cf40b1730 2024-03-25T18:17:46.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.455+0000 7f3cdbfff640 1 -- 172.21.15.162:0/3220436391 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f3ce40522c0 con 0x7f3cf40b1730 2024-03-25T18:17:46.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.455+0000 7f3cdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:46.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.455+0000 7f3cdbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:46.457 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:17:46.457 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:46.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 10 librados: watch_flush enter 2024-03-25T18:17:46.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 10 librados: watch_flush exit 2024-03-25T18:17:46.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T18:17:46.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 10 client.4378.objecter successfully canceled tick 2024-03-25T18:17:46.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 >> v1:172.21.15.191:6800/1552703795 conn(0x7f3ce006e120 legacy=0x7f3ce00705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 10 monclient: shutdown 2024-03-25T18:17:46.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:46.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 >> v1:172.21.15.191:6789/0 conn(0x7f3cf40b1730 legacy=0x7f3cf4134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:46.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 shutdown_connections 2024-03-25T18:17:46.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 >> 172.21.15.162:0/3220436391 conn(0x7f3cf40617a0 msgr2=0x7f3cf40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:46.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 shutdown_connections 2024-03-25T18:17:46.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 1 -- 172.21.15.162:0/3220436391 wait complete. 2024-03-25T18:17:46.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:46.458+0000 7f3cf9d6b640 1 librados: shutdown 2024-03-25T18:17:46.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:53.277 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:53.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.394+0000 7f6be1d32640 10 monclient: get_monmap_and_config 2024-03-25T18:17:53.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.394+0000 7f6be1d32640 10 monclient: build_initial_monmap 2024-03-25T18:17:53.395 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.394+0000 7f6be1d32640 10 monclient: monmap: 2024-03-25T18:17:53.395 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:53.395 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:53.395 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:17:53.395590+0000 2024-03-25T18:17:53.396 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:17:53.395590+0000 2024-03-25T18:17:53.396 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:53.396 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:53.396 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:53.396 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:17:53.396 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:17:53.396 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.395+0000 7f6be1d32640 1 Processor -- start 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.395+0000 7f6be1d32640 1 -- start start 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.395+0000 7f6be1d32640 10 monclient: init 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.395+0000 7f6be1d32640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.395+0000 7f6be1d32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.396+0000 7f6be1d32640 10 monclient(hunting): picked mon.noname-b con 0x7f6bdc0b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.396+0000 7f6be1d32640 10 monclient(hunting): picked mon.noname-c con 0x7f6bdc0b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.396+0000 7f6be1d32640 10 monclient(hunting): picked mon.noname-a con 0x7f6bdc14be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.396+0000 7f6be1d32640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc0af940 con 0x7f6bdc14be90 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.396+0000 7f6be1d32640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc0b0a70 con 0x7f6bdc0b1730 2024-03-25T18:17:53.397 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.396+0000 7f6be1d32640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc0b0d50 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.396+0000 7f6be1d32640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:53.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.396+0000 7f6be1d32640 10 monclient(hunting): authenticate will time out at 7772.669796s 2024-03-25T18:17:53.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.407+0000 7f6bdbfff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6bdc14be90 0x7f6bdc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:41022/0 (socket says 172.21.15.162:41022) 2024-03-25T18:17:53.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.407+0000 7f6bdbfff640 1 -- 172.21.15.162:0/2588302888 learned_addr learned my addr 172.21.15.162:0/2588302888 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:53.422 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.422+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912479684 0 0) 0x7f6bdc0af940 con 0x7f6bdc14be90 2024-03-25T18:17:53.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.422+0000 7f6bda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:53.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.422+0000 7f6bda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:53.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.422+0000 7f6bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:53.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.422+0000 7f6bda7fc640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:17:53.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.422+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc4003520 con 0x7f6bdc14be90 2024-03-25T18:17:53.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.425+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060464755 0 0) 0x7f6bdc0b0a70 con 0x7f6bdc0b1730 2024-03-25T18:17:53.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.425+0000 7f6bda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:53.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.425+0000 7f6bda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:53.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.425+0000 7f6bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:53.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.425+0000 7f6bda7fc640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:17:53.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.425+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bdc0af940 con 0x7f6bdc0b1730 2024-03-25T18:17:53.427 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.426+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748041440 0 0) 0x7f6bdc0b0d50 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.427 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.426+0000 7f6bda7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:53.427 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.426+0000 7f6bda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:53.427 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.426+0000 7f6bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:53.427 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.426+0000 7f6bda7fc640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:17:53.428 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.426+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bdc0b0a70 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.439 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1724898004 0 0) 0x7f6bdc0af940 con 0x7f6bdc0b1730 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 >> v1:172.21.15.162:6790/0 conn(0x7f6bdc0b4d60 legacy=0x7f6bdc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 >> v1:172.21.15.162:6789/0 conn(0x7f6bdc14be90 legacy=0x7f6bdc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 1 monclient: found mon.noname-c 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0b1030 con 0x7f6bdc0b1730 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6bd0001670 con 0x7f6bdc0b1730 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6be1d32640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6be1d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:53.440 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.438+0000 7f6bda7fc640 10 monclient: dump: 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:53.441 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.439+0000 7f6be1d32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:53.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.453+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bd0001890 con 0x7f6bdc0b1730 2024-03-25T18:17:53.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.453+0000 7f6bda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:53.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.453+0000 7f6bda7fc640 1 -- 172.21.15.162:0/2588302888 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6bd0001c20 con 0x7f6bdc0b1730 2024-03-25T18:17:53.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.453+0000 7f6bda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:53.454 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.453+0000 7f6bda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.453+0000 7f6bda7fc640 10 monclient: dump: 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.453+0000 7f6be1d32640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:53.455 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.453+0000 7f6be1d32640 4 set_mon_vals no callback set 2024-03-25T18:17:53.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.454+0000 7f6be1d32640 10 monclient: shutdown 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.454+0000 7f6be1d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.454+0000 7f6be1d32640 1 -- 172.21.15.162:0/2588302888 >> v1:172.21.15.191:6789/0 conn(0x7f6bdc0b1730 legacy=0x7f6bdc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.454+0000 7f6be1d32640 1 -- 172.21.15.162:0/2588302888 shutdown_connections 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.454+0000 7f6be1d32640 1 -- 172.21.15.162:0/2588302888 >> 172.21.15.162:0/2588302888 conn(0x7f6bdc0617a0 msgr2=0x7f6bdc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.454+0000 7f6be1d32640 1 -- 172.21.15.162:0/2588302888 shutdown_connections 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.454+0000 7f6be1d32640 1 -- 172.21.15.162:0/2588302888 wait complete. 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.455+0000 7f6be1d32640 10 monclient: build_initial_monmap 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.455+0000 7f6be1d32640 10 monclient: monmap: 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:17:53.457 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:17:53.458 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:17:53.458 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:17:53.458 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:53.458 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.456+0000 7f6be1d32640 1 librados: starting msgr at 2024-03-25T18:17:53.458 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.456+0000 7f6be1d32640 1 librados: starting objecter 2024-03-25T18:17:53.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.456+0000 7f6be1d32640 1 Processor -- start 2024-03-25T18:17:53.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.456+0000 7f6be1d32640 1 -- start start 2024-03-25T18:17:53.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.456+0000 7f6be1d32640 1 librados: setting wanted keys 2024-03-25T18:17:53.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.456+0000 7f6be1d32640 1 librados: calling monclient init 2024-03-25T18:17:53.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.456+0000 7f6be1d32640 10 monclient: init 2024-03-25T18:17:53.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 10 monclient(hunting): picked mon.noname-a con 0x7f6bdc0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 10 monclient(hunting): picked mon.noname-c con 0x7f6bdc0b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 10 monclient(hunting): picked mon.noname-b con 0x7f6bdc14be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc140380 con 0x7f6bdc0b1730 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc141a70 con 0x7f6bdc14be90 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc13b160 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:53.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.457+0000 7f6be1d32640 10 monclient(hunting): authenticate will time out at 7772.731274s 2024-03-25T18:17:53.464 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.463+0000 7f6bdaffd640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f6bdc0b4d60 0x7f6bdc1d7890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:58920/0 (socket says 172.21.15.162:58920) 2024-03-25T18:17:53.464 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.463+0000 7f6bdaffd640 1 -- 172.21.15.162:0/2773464766 learned_addr learned my addr 172.21.15.162:0/2773464766 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:17:53.466 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.465+0000 7f6bc3fff640 10 client.?.objecter ms_handle_connect 0x7f6bdc0b4d60 2024-03-25T18:17:53.466 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.465+0000 7f6bc3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:53.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.472+0000 7f6bc3fff640 10 client.?.objecter ms_handle_connect 0x7f6bdc0b1730 2024-03-25T18:17:53.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.472+0000 7f6bc3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:53.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.479+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2494905212 0 0) 0x7f6bdc140380 con 0x7f6bdc0b1730 2024-03-25T18:17:53.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.479+0000 7f6bc3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:53.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.479+0000 7f6bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:53.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.479+0000 7f6bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:53.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.479+0000 7f6bc3fff640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:17:53.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.479+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc8004eb0 con 0x7f6bdc0b1730 2024-03-25T18:17:53.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.480+0000 7f6bc3fff640 10 client.?.objecter ms_handle_connect 0x7f6bdc14be90 2024-03-25T18:17:53.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.480+0000 7f6bc3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:53.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.483+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935937244 0 0) 0x7f6bdc13b160 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.483+0000 7f6bc3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:53.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.483+0000 7f6bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:53.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.483+0000 7f6bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:53.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.483+0000 7f6bc3fff640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:17:53.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.483+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc80051d0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577759425 0 0) 0x7f6bdc141a70 con 0x7f6bdc14be90 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bdc13b160 con 0x7f6bdc14be90 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1877633350 0 0) 0x7f6bc8004eb0 con 0x7f6bdc0b1730 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bdc141a70 con 0x7f6bdc0b1730 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6bcc0018f0 con 0x7f6bdc0b1730 2024-03-25T18:17:53.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 10 monclient(hunting): dump: 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:53.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.492+0000 7f6bc3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.500+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 835170444 0 0) 0x7f6bc80051d0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.500+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bc8004eb0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.500+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6bd0001c60 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.500+0000 7f6bc3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.500+0000 7f6bc3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.500+0000 7f6bc3fff640 10 monclient(hunting): dump: 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:53.501 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:53.502 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:53.502 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:53.502 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:53.502 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:53.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 969742824 0 0) 0x7f6bc8004eb0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 >> v1:172.21.15.191:6789/0 conn(0x7f6bdc14be90 legacy=0x7f6bdc1d7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 >> v1:172.21.15.162:6789/0 conn(0x7f6bdc0b1730 legacy=0x7f6bdc134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:53.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 1 monclient: found mon.c 2024-03-25T18:17:53.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc13b9d0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6bc3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 5 monclient: authenticate success, global_id 4388 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 10 monclient: _renew_subs 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc135450 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 10 monclient: _renew_subs 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc1421e0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.517+0000 7f6be1d32640 1 librados: init done 2024-03-25T18:17:53.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.518+0000 7f6be1d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:53.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.518+0000 7f6be1d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:53.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.518+0000 7f6be1d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:53.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.518+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc13bc20 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.836+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bd00038a0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.836+0000 7f6bc3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.836+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6bd0003bf0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.836+0000 7f6bc3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.836+0000 7f6bc3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.836+0000 7f6bc3fff640 10 monclient: dump: 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:17:53.838 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:17:53.839 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:17:53.839 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:17:53.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.836+0000 7f6bd9ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:53.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.837+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f6bd0003e10 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.838+0000 7f6bdb7fe640 10 monclient: get_auth_request con 0x7f6bc806e170 auth_method 0 2024-03-25T18:17:53.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.838+0000 7f6bc3fff640 10 client.4388.objecter ms_handle_connect 0x7f6bc806e170 2024-03-25T18:17:53.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.844+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f6bd00548a0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.844+0000 7f6bc3fff640 10 client.4388.objecter ms_dispatch 0x7f6bdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:53.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.844+0000 7f6bc3fff640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:53.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.844+0000 7f6bc3fff640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:53.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.844+0000 7f6bc3fff640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T18:17:53.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.844+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6bd00503b0 con 0x7f6bdc0b4d60 2024-03-25T18:17:53.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.844+0000 7f6bc3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:53.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:53.844+0000 7f6bc3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:54.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.094+0000 7f6be1d32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:54.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.094+0000 7f6be1d32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:54.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.094+0000 7f6be1d32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:17:54.096 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.094+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bdc0627b0 con 0x7f6bdc0b4d60 2024-03-25T18:17:54.116 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.115+0000 7f6bc3fff640 1 -- 172.21.15.162:0/2773464766 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f6bd00522e0 con 0x7f6bdc0b4d60 2024-03-25T18:17:54.116 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.115+0000 7f6bc3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:54.116 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.115+0000 7f6bc3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:54.116 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:17:54.117 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:54.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 10 librados: watch_flush enter 2024-03-25T18:17:54.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 10 librados: watch_flush exit 2024-03-25T18:17:54.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T18:17:54.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 10 client.4388.objecter successfully canceled tick 2024-03-25T18:17:54.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 >> v1:172.21.15.191:6800/1552703795 conn(0x7f6bc806e170 legacy=0x7f6bc8070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 10 monclient: shutdown 2024-03-25T18:17:54.119 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:54.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 >> v1:172.21.15.162:6790/0 conn(0x7f6bdc0b4d60 legacy=0x7f6bdc1d7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 shutdown_connections 2024-03-25T18:17:54.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 >> 172.21.15.162:0/2773464766 conn(0x7f6bdc0617a0 msgr2=0x7f6bdc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:54.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 shutdown_connections 2024-03-25T18:17:54.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 1 -- 172.21.15.162:0/2773464766 wait complete. 2024-03-25T18:17:54.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:17:54.118+0000 7f6be1d32640 1 librados: shutdown 2024-03-25T18:17:54.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:00.938 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:01.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.050+0000 7f7833132640 10 monclient: get_monmap_and_config 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.050+0000 7f7833132640 10 monclient: build_initial_monmap 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.050+0000 7f7833132640 10 monclient: monmap: 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:18:01.051794+0000 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:18:01.051794+0000 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:01.052 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:18:01.053 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:18:01.053 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:01.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.051+0000 7f7833132640 1 Processor -- start 2024-03-25T18:18:01.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.051+0000 7f7833132640 1 -- start start 2024-03-25T18:18:01.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.051+0000 7f7833132640 10 monclient: init 2024-03-25T18:18:01.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:01.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:01.054 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 10 monclient(hunting): picked mon.noname-a con 0x7f782c0a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:01.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 10 monclient(hunting): picked mon.noname-c con 0x7f782c13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:01.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 10 monclient(hunting): picked mon.noname-b con 0x7f782c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:01.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c142520 con 0x7f782c0a98f0 2024-03-25T18:18:01.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c0b28a0 con 0x7f782c13f710 2024-03-25T18:18:01.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c0b2a30 con 0x7f782c1430c0 2024-03-25T18:18:01.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:01.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.052+0000 7f7833132640 10 monclient(hunting): authenticate will time out at 7780.326470s 2024-03-25T18:18:01.069 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.068+0000 7f782bfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f782c13f710 0x7f782c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:39508/0 (socket says 172.21.15.162:39508) 2024-03-25T18:18:01.069 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.068+0000 7f782bfff640 1 -- 172.21.15.162:0/4222546395 learned_addr learned my addr 172.21.15.162:0/4222546395 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:01.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.084+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937053125 0 0) 0x7f782c142520 con 0x7f782c0a98f0 2024-03-25T18:18:01.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.085+0000 7f782b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.085+0000 7f782b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.085+0000 7f782b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.085+0000 7f782b7fe640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:18:01.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.085+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f781c003520 con 0x7f782c0a98f0 2024-03-25T18:18:01.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.089+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185256952 0 0) 0x7f782c0b28a0 con 0x7f782c13f710 2024-03-25T18:18:01.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.089+0000 7f782b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.089+0000 7f782b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.089+0000 7f782b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.089+0000 7f782b7fe640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:18:01.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.089+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f782c142520 con 0x7f782c13f710 2024-03-25T18:18:01.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.096+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73001796 0 0) 0x7f781c003520 con 0x7f782c0a98f0 2024-03-25T18:18:01.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 >> v1:172.21.15.162:6790/0 conn(0x7f782c1430c0 legacy=0x7f782c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 >> v1:172.21.15.191:6789/0 conn(0x7f782c13f710 legacy=0x7f782c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:01.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:18:01.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:01.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:18:01.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c0b44c0 con 0x7f782c0a98f0 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7820001690 con 0x7f782c0a98f0 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f7833132640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f7833132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f782b7fe640 10 monclient: dump: 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:01.099 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:01.100 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:01.100 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:01.100 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:01.100 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:01.100 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:01.100 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.097+0000 7f7833132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:01.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.106+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78200017f0 con 0x7f782c0a98f0 2024-03-25T18:18:01.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.106+0000 7f782b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.106+0000 7f782b7fe640 1 -- 172.21.15.162:0/4222546395 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7820001c00 con 0x7f782c0a98f0 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.106+0000 7f7833132640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.106+0000 7f7833132640 4 set_mon_vals no callback set 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.106+0000 7f782b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.106+0000 7f782b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.106+0000 7f782b7fe640 10 monclient: dump: 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:01.108 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:01.109 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:01.109 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:01.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.107+0000 7f7833132640 10 monclient: shutdown 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.107+0000 7f7833132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.107+0000 7f7833132640 1 -- 172.21.15.162:0/4222546395 >> v1:172.21.15.162:6789/0 conn(0x7f782c0a98f0 legacy=0x7f782c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.107+0000 7f7833132640 1 -- 172.21.15.162:0/4222546395 shutdown_connections 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.107+0000 7f7833132640 1 -- 172.21.15.162:0/4222546395 >> 172.21.15.162:0/4222546395 conn(0x7f782c0b7ea0 msgr2=0x7f782c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.107+0000 7f7833132640 1 -- 172.21.15.162:0/4222546395 shutdown_connections 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.108+0000 7f7833132640 1 -- 172.21.15.162:0/4222546395 wait complete. 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 10 monclient: build_initial_monmap 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 10 monclient: monmap: 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:18:01.110 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:01.111 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:18:01.111 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:18:01.111 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 1 librados: starting msgr at 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 1 librados: starting objecter 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 1 Processor -- start 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 1 -- start start 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 1 librados: setting wanted keys 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 1 librados: calling monclient init 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.109+0000 7f7833132640 10 monclient: init 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.110+0000 7f7833132640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.110+0000 7f7833132640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:01.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.110+0000 7f7833132640 10 monclient(hunting): picked mon.noname-b con 0x7f782c0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:01.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.110+0000 7f7833132640 10 monclient(hunting): picked mon.noname-a con 0x7f782c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:01.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.110+0000 7f7833132640 10 monclient(hunting): picked mon.noname-c con 0x7f782c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:01.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.110+0000 7f7833132640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c0b7ce0 con 0x7f782c13f710 2024-03-25T18:18:01.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.111+0000 7f7833132640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c1319b0 con 0x7f782c0a98f0 2024-03-25T18:18:01.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.111+0000 7f7833132640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c0b7800 con 0x7f782c1430c0 2024-03-25T18:18:01.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.111+0000 7f7833132640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:01.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.111+0000 7f7833132640 10 monclient(hunting): authenticate will time out at 7780.384761s 2024-03-25T18:18:01.122 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.122+0000 7f7830ece640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f782c0a98f0 0x7f782c1dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:39572/0 (socket says 172.21.15.162:39572) 2024-03-25T18:18:01.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.122+0000 7f7830ece640 1 -- 172.21.15.162:0/547672234 learned_addr learned my addr 172.21.15.162:0/547672234 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:01.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.123+0000 7f78297fa640 10 client.?.objecter ms_handle_connect 0x7f782c0a98f0 2024-03-25T18:18:01.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.123+0000 7f78297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:01.129 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.128+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982989859 0 0) 0x7f782c1319b0 con 0x7f782c0a98f0 2024-03-25T18:18:01.129 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.128+0000 7f78297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.129 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.128+0000 7f78297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.129 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.128+0000 7f78297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.129 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.128+0000 7f78297fa640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:18:01.129 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.128+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7810004eb0 con 0x7f782c0a98f0 2024-03-25T18:18:01.131 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.130+0000 7f78297fa640 10 client.?.objecter ms_handle_connect 0x7f782c13f710 2024-03-25T18:18:01.131 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.130+0000 7f78297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:01.132 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.131+0000 7f78297fa640 10 client.?.objecter ms_handle_connect 0x7f782c1430c0 2024-03-25T18:18:01.132 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.131+0000 7f78297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:01.139 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.138+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720883737 0 0) 0x7f782c0b7ce0 con 0x7f782c13f710 2024-03-25T18:18:01.139 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.138+0000 7f78297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.139 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.138+0000 7f78297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.138+0000 7f78297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.138+0000 7f78297fa640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:18:01.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.138+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f782c1319b0 con 0x7f782c13f710 2024-03-25T18:18:01.141 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.140+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2945800409 0 0) 0x7f7810004eb0 con 0x7f782c0a98f0 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.140+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f782c0b7ce0 con 0x7f782c0a98f0 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.140+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7820001690 con 0x7f782c0a98f0 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.140+0000 7f78297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.140+0000 7f78297fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.140+0000 7f78297fa640 10 monclient(hunting): dump: 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:01.142 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:01.143 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:01.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.140+0000 7f78297fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:01.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.144+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256049173 0 0) 0x7f782c0b7800 con 0x7f782c1430c0 2024-03-25T18:18:01.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.144+0000 7f78297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:01.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.144+0000 7f78297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:01.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.144+0000 7f78297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:01.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.144+0000 7f78297fa640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:18:01.145 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.144+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7810004eb0 con 0x7f782c1430c0 2024-03-25T18:18:01.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4248080835 0 0) 0x7f782c0b7ce0 con 0x7f782c0a98f0 2024-03-25T18:18:01.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 >> v1:172.21.15.162:6790/0 conn(0x7f782c1430c0 legacy=0x7f782c1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 >> v1:172.21.15.162:6789/0 conn(0x7f782c13f710 legacy=0x7f782c130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 1 monclient: found mon.b 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c136f50 con 0x7f782c0a98f0 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f78297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 5 monclient: authenticate success, global_id 4390 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 10 monclient: _renew_subs 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c1350f0 con 0x7f782c0a98f0 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 10 monclient: _renew_subs 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:01.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c135c60 con 0x7f782c0a98f0 2024-03-25T18:18:01.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.151+0000 7f7833132640 1 librados: init done 2024-03-25T18:18:01.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.152+0000 7f7833132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:01.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.152+0000 7f7833132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:01.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.152+0000 7f7833132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:01.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.152+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c0627b0 con 0x7f782c0a98f0 2024-03-25T18:18:01.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.424+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f78200038a0 con 0x7f782c0a98f0 2024-03-25T18:18:01.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.424+0000 7f78297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:01.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.424+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7820003c70 con 0x7f782c0a98f0 2024-03-25T18:18:01.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.424+0000 7f78297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:01.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.424+0000 7f78297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:01.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.424+0000 7f78297fa640 10 monclient: dump: 2024-03-25T18:18:01.425 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:01.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.424+0000 7f782affd640 4 set_mon_vals no callback set 2024-03-25T18:18:01.427 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.424+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f7820003e90 con 0x7f782c0a98f0 2024-03-25T18:18:01.427 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.425+0000 7f782bfff640 10 monclient: get_auth_request con 0x7f781006de00 auth_method 0 2024-03-25T18:18:01.427 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.425+0000 7f78297fa640 10 client.4390.objecter ms_handle_connect 0x7f781006de00 2024-03-25T18:18:01.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.430+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f7820054910 con 0x7f782c0a98f0 2024-03-25T18:18:01.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.430+0000 7f78297fa640 10 client.4390.objecter ms_dispatch 0x7f782c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:01.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.430+0000 7f78297fa640 3 client.4390.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:01.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.430+0000 7f78297fa640 3 client.4390.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:01.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.430+0000 7f78297fa640 20 client.4390.objecter dump_active .. 0 homeless 2024-03-25T18:18:01.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.431+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7820050420 con 0x7f782c0a98f0 2024-03-25T18:18:01.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.431+0000 7f78297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:01.432 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.431+0000 7f78297fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:01.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.667+0000 7f7833132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:01.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.667+0000 7f7833132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:01.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.667+0000 7f7833132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:01.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.667+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f782c13e270 con 0x7f782c0a98f0 2024-03-25T18:18:01.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.755+0000 7f78297fa640 1 -- 172.21.15.162:0/547672234 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f7820052350 con 0x7f782c0a98f0 2024-03-25T18:18:01.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.755+0000 7f78297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:01.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.755+0000 7f78297fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:01.757 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:18:01.757 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:01.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 10 librados: watch_flush enter 2024-03-25T18:18:01.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 10 librados: watch_flush exit 2024-03-25T18:18:01.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 20 client.4390.objecter shutdown clearing up homeless session... 2024-03-25T18:18:01.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 10 client.4390.objecter successfully canceled tick 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 >> v1:172.21.15.191:6800/1552703795 conn(0x7f781006de00 legacy=0x7f78100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 10 monclient: shutdown 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 >> v1:172.21.15.191:6789/0 conn(0x7f782c0a98f0 legacy=0x7f782c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 shutdown_connections 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 >> 172.21.15.162:0/547672234 conn(0x7f782c0b7ea0 msgr2=0x7f782c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 shutdown_connections 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 1 -- 172.21.15.162:0/547672234 wait complete. 2024-03-25T18:18:01.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:01.758+0000 7f7833132640 1 librados: shutdown 2024-03-25T18:18:01.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:08.577 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:08.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.673+0000 7fdbf6d1c640 10 monclient: get_monmap_and_config 2024-03-25T18:18:08.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.673+0000 7fdbf6d1c640 10 monclient: build_initial_monmap 2024-03-25T18:18:08.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.673+0000 7fdbf6d1c640 10 monclient: monmap: 2024-03-25T18:18:08.674 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:18:08.674829+0000 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:18:08.674829+0000 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:18:08.675 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:08.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.674+0000 7fdbf6d1c640 1 Processor -- start 2024-03-25T18:18:08.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.674+0000 7fdbf6d1c640 1 -- start start 2024-03-25T18:18:08.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.674+0000 7fdbf6d1c640 10 monclient: init 2024-03-25T18:18:08.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:08.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:08.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 10 monclient(hunting): picked mon.noname-b con 0x7fdbf00b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:08.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 10 monclient(hunting): picked mon.noname-c con 0x7fdbf00b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:08.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 10 monclient(hunting): picked mon.noname-a con 0x7fdbf014be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:08.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf00af940 con 0x7fdbf014be90 2024-03-25T18:18:08.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf00b0a70 con 0x7fdbf00b1730 2024-03-25T18:18:08.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf00b0d50 con 0x7fdbf00b4d60 2024-03-25T18:18:08.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:08.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.675+0000 7fdbf6d1c640 10 monclient(hunting): authenticate will time out at 7787.948867s 2024-03-25T18:18:08.695 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.694+0000 7fdbf52b9640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fdbf014be90 0x7fdbf014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38304/0 (socket says 172.21.15.162:38304) 2024-03-25T18:18:08.695 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.694+0000 7fdbf52b9640 1 -- 172.21.15.162:0/932422514 learned_addr learned my addr 172.21.15.162:0/932422514 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:08.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.719+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621438382 0 0) 0x7fdbf00af940 con 0x7fdbf014be90 2024-03-25T18:18:08.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.719+0000 7fdbef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:08.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.719+0000 7fdbef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:08.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.719+0000 7fdbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:08.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.719+0000 7fdbef7fe640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:18:08.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.719+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbe0003520 con 0x7fdbf014be90 2024-03-25T18:18:08.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.721+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422476278 0 0) 0x7fdbf00b0a70 con 0x7fdbf00b1730 2024-03-25T18:18:08.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.721+0000 7fdbef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:08.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.721+0000 7fdbef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:08.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.721+0000 7fdbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:08.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.721+0000 7fdbef7fe640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:18:08.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.721+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf00af940 con 0x7fdbf00b1730 2024-03-25T18:18:08.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.728+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945409497 0 0) 0x7fdbf00b0d50 con 0x7fdbf00b4d60 2024-03-25T18:18:08.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.728+0000 7fdbef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:08.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.728+0000 7fdbef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:08.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.728+0000 7fdbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:08.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.728+0000 7fdbef7fe640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:18:08.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.728+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf00b0a70 con 0x7fdbf00b4d60 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 16685773 0 0) 0x7fdbe0003520 con 0x7fdbf014be90 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 >> v1:172.21.15.162:6790/0 conn(0x7fdbf00b4d60 legacy=0x7fdbf013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 >> v1:172.21.15.191:6789/0 conn(0x7fdbf00b1730 legacy=0x7fdbf00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 1 monclient: found mon.noname-a 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf00b1030 con 0x7fdbf014be90 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbf6d1c640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:18:08.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.731+0000 7fdbf6d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:08.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.732+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdbe80019d0 con 0x7fdbf014be90 2024-03-25T18:18:08.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.732+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:08.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.732+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:08.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.732+0000 7fdbef7fe640 10 monclient: dump: 2024-03-25T18:18:08.733 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:08.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.732+0000 7fdbf6d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:08.765 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.764+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbe8001c20 con 0x7fdbf014be90 2024-03-25T18:18:08.765 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.764+0000 7fdbef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:08.765 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.764+0000 7fdbef7fe640 1 -- 172.21.15.162:0/932422514 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdbe8001fb0 con 0x7fdbf014be90 2024-03-25T18:18:08.765 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.764+0000 7fdbf6d1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.764+0000 7fdbf6d1c640 4 set_mon_vals no callback set 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.764+0000 7fdbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.764+0000 7fdbef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.764+0000 7fdbef7fe640 10 monclient: dump: 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:08.766 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.765+0000 7fdbf6d1c640 10 monclient: shutdown 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.765+0000 7fdbf6d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.765+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/932422514 >> v1:172.21.15.162:6789/0 conn(0x7fdbf014be90 legacy=0x7fdbf014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.765+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/932422514 shutdown_connections 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.765+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/932422514 >> 172.21.15.162:0/932422514 conn(0x7fdbf00617a0 msgr2=0x7fdbf0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.765+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/932422514 shutdown_connections 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.765+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/932422514 wait complete. 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.766+0000 7fdbf6d1c640 10 monclient: build_initial_monmap 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.766+0000 7fdbf6d1c640 10 monclient: monmap: 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:08.768 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.766+0000 7fdbf6d1c640 1 librados: starting msgr at 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.766+0000 7fdbf6d1c640 1 librados: starting objecter 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.766+0000 7fdbf6d1c640 1 Processor -- start 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.767+0000 7fdbf6d1c640 1 -- start start 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.767+0000 7fdbf6d1c640 1 librados: setting wanted keys 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.767+0000 7fdbf6d1c640 1 librados: calling monclient init 2024-03-25T18:18:08.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.767+0000 7fdbf6d1c640 10 monclient: init 2024-03-25T18:18:08.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.767+0000 7fdbf6d1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:08.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.767+0000 7fdbf6d1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:08.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.767+0000 7fdbf6d1c640 10 monclient(hunting): picked mon.noname-a con 0x7fdbf00b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:08.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.767+0000 7fdbf6d1c640 10 monclient(hunting): picked mon.noname-b con 0x7fdbf00b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:08.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.768+0000 7fdbf6d1c640 10 monclient(hunting): picked mon.noname-c con 0x7fdbf014be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:08.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.768+0000 7fdbf6d1c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf0138740 con 0x7fdbf00b1730 2024-03-25T18:18:08.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.768+0000 7fdbf6d1c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf01388d0 con 0x7fdbf00b4d60 2024-03-25T18:18:08.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.768+0000 7fdbf6d1c640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf0138dd0 con 0x7fdbf014be90 2024-03-25T18:18:08.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.768+0000 7fdbf6d1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:08.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.768+0000 7fdbf6d1c640 10 monclient(hunting): authenticate will time out at 7788.041781s 2024-03-25T18:18:08.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.776+0000 7fdbf4ab8640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fdbf00b1730 0x7fdbf0131cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38370/0 (socket says 172.21.15.162:38370) 2024-03-25T18:18:08.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.777+0000 7fdbf4ab8640 1 -- 172.21.15.162:0/52076629 learned_addr learned my addr 172.21.15.162:0/52076629 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:08.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.780+0000 7fdbed7fa640 10 client.?.objecter ms_handle_connect 0x7fdbf00b1730 2024-03-25T18:18:08.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.780+0000 7fdbed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:08.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.781+0000 7fdbed7fa640 10 client.?.objecter ms_handle_connect 0x7fdbf014be90 2024-03-25T18:18:08.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.781+0000 7fdbed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:08.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.794+0000 7fdbed7fa640 10 client.?.objecter ms_handle_connect 0x7fdbf00b4d60 2024-03-25T18:18:08.795 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.794+0000 7fdbed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:08.799 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.799+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797380129 0 0) 0x7fdbf0138740 con 0x7fdbf00b1730 2024-03-25T18:18:08.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.799+0000 7fdbed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:08.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.799+0000 7fdbed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:08.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.799+0000 7fdbed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:08.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.799+0000 7fdbed7fa640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:18:08.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.799+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbd4004eb0 con 0x7fdbf00b1730 2024-03-25T18:18:08.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.800+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790619154 0 0) 0x7fdbf0138dd0 con 0x7fdbf014be90 2024-03-25T18:18:08.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.800+0000 7fdbed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:08.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.800+0000 7fdbed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:08.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.800+0000 7fdbed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:08.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.800+0000 7fdbed7fa640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:18:08.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.800+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf0138740 con 0x7fdbf014be90 2024-03-25T18:18:08.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.808+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318105600 0 0) 0x7fdbf01388d0 con 0x7fdbf00b4d60 2024-03-25T18:18:08.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.808+0000 7fdbed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:08.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.808+0000 7fdbed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:08.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.808+0000 7fdbed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:08.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.808+0000 7fdbed7fa640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:18:08.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.808+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf0138dd0 con 0x7fdbf00b4d60 2024-03-25T18:18:08.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.813+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278803859 0 0) 0x7fdbf0138740 con 0x7fdbf014be90 2024-03-25T18:18:08.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.813+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbf01388d0 con 0x7fdbf014be90 2024-03-25T18:18:08.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.813+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdbe8001f30 con 0x7fdbf014be90 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.813+0000 7fdbed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.813+0000 7fdbed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.813+0000 7fdbed7fa640 10 monclient(hunting): dump: 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:08.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.813+0000 7fdbed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:08.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.816+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3319326037 0 0) 0x7fdbf0138dd0 con 0x7fdbf00b4d60 2024-03-25T18:18:08.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.816+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbf0138740 con 0x7fdbf00b4d60 2024-03-25T18:18:08.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.816+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdbdc001870 con 0x7fdbf00b4d60 2024-03-25T18:18:08.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.816+0000 7fdbed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:08.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.816+0000 7fdbed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:08.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.816+0000 7fdbed7fa640 10 monclient(hunting): dump: 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:08.818 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.817+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3430036403 0 0) 0x7fdbd4004eb0 con 0x7fdbf00b1730 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.817+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbf0138dd0 con 0x7fdbf00b1730 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.817+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdbe4001a10 con 0x7fdbf00b1730 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.817+0000 7fdbed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.817+0000 7fdbed7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.817+0000 7fdbed7fa640 10 monclient(hunting): dump: 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:08.819 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:08.820 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:08.820 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:08.826 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 24252961 0 0) 0x7fdbf0138740 con 0x7fdbf00b4d60 2024-03-25T18:18:08.826 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 >> v1:172.21.15.162:6790/0 conn(0x7fdbf014be90 legacy=0x7fdbf01db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.826 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 >> v1:172.21.15.162:6789/0 conn(0x7fdbf00b1730 legacy=0x7fdbf0131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 1 monclient: found mon.b 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf0138f60 con 0x7fdbf00b4d60 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 10 monclient: _renew_subs 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf0137da0 con 0x7fdbf00b4d60 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 10 monclient: _renew_subs 2024-03-25T18:18:08.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:08.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf01451e0 con 0x7fdbf00b4d60 2024-03-25T18:18:08.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.825+0000 7fdbf6d1c640 1 librados: init done 2024-03-25T18:18:08.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.826+0000 7fdbf6d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:08.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.826+0000 7fdbf6d1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:08.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.826+0000 7fdbf6d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:08.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:08.826+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf0139800 con 0x7fdbf00b4d60 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.104+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdbdc001ce0 con 0x7fdbf00b4d60 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.104+0000 7fdbed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.104+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdbdc003790 con 0x7fdbf00b4d60 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.104+0000 7fdbed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.104+0000 7fdbed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.104+0000 7fdbed7fa640 10 monclient: dump: 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:09.106 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:09.107 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:09.107 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:09.107 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:09.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.104+0000 7fdbeeffd640 4 set_mon_vals no callback set 2024-03-25T18:18:09.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.105+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fdbdc0197a0 con 0x7fdbf00b4d60 2024-03-25T18:18:09.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.106+0000 7fdbf4ab8640 10 monclient: get_auth_request con 0x7fdbd406e0e0 auth_method 0 2024-03-25T18:18:09.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.106+0000 7fdbed7fa640 10 client.4396.objecter ms_handle_connect 0x7fdbd406e0e0 2024-03-25T18:18:09.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.108+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fdbdc0534e0 con 0x7fdbf00b4d60 2024-03-25T18:18:09.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.108+0000 7fdbed7fa640 10 client.4396.objecter ms_dispatch 0x7fdbf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:09.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.108+0000 7fdbed7fa640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:09.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.108+0000 7fdbed7fa640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:09.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.108+0000 7fdbed7fa640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T18:18:09.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.114+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdbdc04ee20 con 0x7fdbf00b4d60 2024-03-25T18:18:09.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.114+0000 7fdbed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:09.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.114+0000 7fdbed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:09.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.371+0000 7fdbf6d1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:09.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.371+0000 7fdbf6d1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:09.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.371+0000 7fdbf6d1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:09.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.371+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbf00627b0 con 0x7fdbf00b4d60 2024-03-25T18:18:09.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.395+0000 7fdbed7fa640 1 -- 172.21.15.162:0/52076629 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fdbdc050f20 con 0x7fdbf00b4d60 2024-03-25T18:18:09.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.395+0000 7fdbed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:09.396 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.395+0000 7fdbed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:09.396 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:18:09.396 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 10 librados: watch_flush enter 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 10 librados: watch_flush exit 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 10 client.4396.objecter successfully canceled tick 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 >> v1:172.21.15.191:6800/1552703795 conn(0x7fdbd406e0e0 legacy=0x7fdbd40705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 10 monclient: shutdown 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 >> v1:172.21.15.191:6789/0 conn(0x7fdbf00b4d60 legacy=0x7fdbf01daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:09.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 shutdown_connections 2024-03-25T18:18:09.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 >> 172.21.15.162:0/52076629 conn(0x7fdbf00617a0 msgr2=0x7fdbf00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:09.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 shutdown_connections 2024-03-25T18:18:09.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 1 -- 172.21.15.162:0/52076629 wait complete. 2024-03-25T18:18:09.401 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:09.398+0000 7fdbf6d1c640 1 librados: shutdown 2024-03-25T18:18:09.410 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:16.216 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:16.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.331+0000 7f085c382640 10 monclient: get_monmap_and_config 2024-03-25T18:18:16.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.331+0000 7f085c382640 10 monclient: build_initial_monmap 2024-03-25T18:18:16.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.331+0000 7f085c382640 10 monclient: monmap: 2024-03-25T18:18:16.332 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:16.332 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:16.332 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:18:16.332495+0000 2024-03-25T18:18:16.332 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:18:16.332495+0000 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.332+0000 7f085c382640 1 Processor -- start 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.332+0000 7f085c382640 1 -- start start 2024-03-25T18:18:16.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.332+0000 7f085c382640 10 monclient: init 2024-03-25T18:18:16.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:16.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:16.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 10 monclient(hunting): picked mon.noname-c con 0x7f08540a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:16.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 10 monclient(hunting): picked mon.noname-b con 0x7f085413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:16.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 10 monclient(hunting): picked mon.noname-a con 0x7f08541430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:16.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0854142520 con 0x7f08541430c0 2024-03-25T18:18:16.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08540b28a0 con 0x7f08540a98f0 2024-03-25T18:18:16.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08540b2a30 con 0x7f085413f710 2024-03-25T18:18:16.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:16.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.333+0000 7f085c382640 10 monclient(hunting): authenticate will time out at 7795.607565s 2024-03-25T18:18:16.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.344+0000 7f085a91f640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f08541430c0 0x7f085414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:54114/0 (socket says 172.21.15.162:54114) 2024-03-25T18:18:16.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.345+0000 7f085a91f640 1 -- 172.21.15.162:0/3528911412 learned_addr learned my addr 172.21.15.162:0/3528911412 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:16.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.361+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162958075 0 0) 0x7f08540b28a0 con 0x7f08540a98f0 2024-03-25T18:18:16.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.361+0000 7f085911c640 10 monclient(hunting): _negotiate 2024-03-25T18:18:16.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.361+0000 7f085911c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:16.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.361+0000 7f085911c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:16.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.361+0000 7f085911c640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:18:16.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.361+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0840003520 con 0x7f08540a98f0 2024-03-25T18:18:16.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.366+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2773961304 0 0) 0x7f0854142520 con 0x7f08541430c0 2024-03-25T18:18:16.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.366+0000 7f085911c640 10 monclient(hunting): _negotiate 2024-03-25T18:18:16.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.366+0000 7f085911c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:16.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.366+0000 7f085911c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:16.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.366+0000 7f085911c640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:18:16.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.366+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08540b28a0 con 0x7f08541430c0 2024-03-25T18:18:16.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.373+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801727119 0 0) 0x7f08540b2a30 con 0x7f085413f710 2024-03-25T18:18:16.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.373+0000 7f085911c640 10 monclient(hunting): _negotiate 2024-03-25T18:18:16.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.373+0000 7f085911c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:16.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.373+0000 7f085911c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:16.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.373+0000 7f085911c640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:18:16.374 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.373+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0854142520 con 0x7f085413f710 2024-03-25T18:18:16.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1641681766 0 0) 0x7f0854142520 con 0x7f085413f710 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 >> v1:172.21.15.191:6789/0 conn(0x7f08540a98f0 legacy=0x7f08540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 >> v1:172.21.15.162:6789/0 conn(0x7f08541430c0 legacy=0x7f085414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 10 monclient: _finish_hunting 0 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 1 monclient: found mon.noname-b 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08540b44c0 con 0x7f085413f710 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 10 monclient: _finish_auth 0 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 10 monclient: _check_auth_tickets 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f08440015b0 con 0x7f085413f710 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085c382640 5 monclient: authenticate success, global_id 4403 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085c382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:16.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085911c640 10 monclient: dump: 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:16.387 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.384+0000 7f085c382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:16.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.402+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0844001710 con 0x7f085413f710 2024-03-25T18:18:16.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.402+0000 7f085911c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:16.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.402+0000 7f085911c640 1 -- 172.21.15.162:0/3528911412 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0844001b60 con 0x7f085413f710 2024-03-25T18:18:16.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.402+0000 7f085911c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.402+0000 7f085911c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.402+0000 7f085911c640 10 monclient: dump: 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.402+0000 7f085c382640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:16.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.402+0000 7f085c382640 4 set_mon_vals no callback set 2024-03-25T18:18:16.406 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.403+0000 7f085c382640 10 monclient: shutdown 2024-03-25T18:18:16.406 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.403+0000 7f085c382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:16.406 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.403+0000 7f085c382640 1 -- 172.21.15.162:0/3528911412 >> v1:172.21.15.162:6790/0 conn(0x7f085413f710 legacy=0x7f08541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.406 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.403+0000 7f085c382640 1 -- 172.21.15.162:0/3528911412 shutdown_connections 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.403+0000 7f085c382640 1 -- 172.21.15.162:0/3528911412 >> 172.21.15.162:0/3528911412 conn(0x7f08540b7ea0 msgr2=0x7f08540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.403+0000 7f085c382640 1 -- 172.21.15.162:0/3528911412 shutdown_connections 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.403+0000 7f085c382640 1 -- 172.21.15.162:0/3528911412 wait complete. 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 10 monclient: build_initial_monmap 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 10 monclient: monmap: 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 1 librados: starting msgr at 2024-03-25T18:18:16.407 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 1 librados: starting objecter 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 1 Processor -- start 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 1 -- start start 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 1 librados: setting wanted keys 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 1 librados: calling monclient init 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.404+0000 7f085c382640 10 monclient: init 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 10 monclient(hunting): picked mon.noname-a con 0x7f08540a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 10 monclient(hunting): picked mon.noname-c con 0x7f085413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 10 monclient(hunting): picked mon.noname-b con 0x7f08541430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08540aef10 con 0x7f08540a98f0 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08540ad9c0 con 0x7f08541430c0 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0854138c20 con 0x7f085413f710 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:16.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.405+0000 7f085c382640 10 monclient(hunting): authenticate will time out at 7795.679389s 2024-03-25T18:18:16.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.410+0000 7f085991d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f085413f710 0x7f08540ad2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43236/0 (socket says 172.21.15.162:43236) 2024-03-25T18:18:16.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.410+0000 7f085991d640 1 -- 172.21.15.162:0/370982945 learned_addr learned my addr 172.21.15.162:0/370982945 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:16.415 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.414+0000 7f084affd640 10 client.?.objecter ms_handle_connect 0x7f08541430c0 2024-03-25T18:18:16.415 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.414+0000 7f084affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:16.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.416+0000 7f084affd640 10 client.?.objecter ms_handle_connect 0x7f08540a98f0 2024-03-25T18:18:16.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.416+0000 7f084affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:16.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.417+0000 7f084affd640 10 client.?.objecter ms_handle_connect 0x7f085413f710 2024-03-25T18:18:16.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.417+0000 7f084affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:16.422 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.421+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366101658 0 0) 0x7f08540ad9c0 con 0x7f08541430c0 2024-03-25T18:18:16.422 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.421+0000 7f084affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:16.422 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.421+0000 7f084affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:16.422 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.421+0000 7f084affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:16.422 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.421+0000 7f084affd640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:18:16.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.421+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0838004eb0 con 0x7f08541430c0 2024-03-25T18:18:16.423 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.423+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911990509 0 0) 0x7f08540aef10 con 0x7f08540a98f0 2024-03-25T18:18:16.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.423+0000 7f084affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:16.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.423+0000 7f084affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:16.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.423+0000 7f084affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:16.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.423+0000 7f084affd640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:18:16.424 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.423+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08540ad9c0 con 0x7f08540a98f0 2024-03-25T18:18:16.429 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.428+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795948689 0 0) 0x7f0854138c20 con 0x7f085413f710 2024-03-25T18:18:16.429 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.428+0000 7f084affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:16.429 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.428+0000 7f084affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:16.429 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.428+0000 7f084affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:16.429 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.428+0000 7f084affd640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:18:16.429 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.428+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08540aef10 con 0x7f085413f710 2024-03-25T18:18:16.433 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.432+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1492384367 0 0) 0x7f0838004eb0 con 0x7f08541430c0 2024-03-25T18:18:16.433 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.432+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0854138c20 con 0x7f08541430c0 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.433+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f084c001c70 con 0x7f08541430c0 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.433+0000 7f084affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.433+0000 7f084affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.433+0000 7f084affd640 10 monclient(hunting): dump: 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:16.434 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:16.435 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:16.435 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:16.435 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:16.435 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:16.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.433+0000 7f084affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:16.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.436+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024127036 0 0) 0x7f08540ad9c0 con 0x7f08540a98f0 2024-03-25T18:18:16.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.436+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0838004eb0 con 0x7f08540a98f0 2024-03-25T18:18:16.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.436+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f0850001950 con 0x7f08540a98f0 2024-03-25T18:18:16.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.436+0000 7f084affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.436+0000 7f084affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.436+0000 7f084affd640 10 monclient(hunting): dump: 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:16.438 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:16.446 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1008465282 0 0) 0x7f0854138c20 con 0x7f08541430c0 2024-03-25T18:18:16.446 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 >> v1:172.21.15.162:6790/0 conn(0x7f085413f710 legacy=0x7f08540ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 >> v1:172.21.15.162:6789/0 conn(0x7f08540a98f0 legacy=0x7f0854137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 1 monclient: found mon.b 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08540b0a10 con 0x7f08541430c0 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 10 monclient: _finish_auth 0 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f084affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 5 monclient: authenticate success, global_id 4402 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 10 monclient: _renew_subs 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08540b0cf0 con 0x7f08541430c0 2024-03-25T18:18:16.447 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:16.448 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 10 monclient: _renew_subs 2024-03-25T18:18:16.448 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:16.448 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08540b12b0 con 0x7f08541430c0 2024-03-25T18:18:16.448 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.445+0000 7f085c382640 1 librados: init done 2024-03-25T18:18:16.448 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.446+0000 7f085c382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:16.448 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.446+0000 7f085c382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:16.448 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.446+0000 7f085c382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:16.449 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.446+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08540627b0 con 0x7f08541430c0 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.719+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f084c004b60 con 0x7f08541430c0 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.719+0000 7f084affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.719+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f084c002100 con 0x7f08541430c0 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.719+0000 7f084affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.719+0000 7f084affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.719+0000 7f084affd640 10 monclient: dump: 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:16.720 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:16.721 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:16.721 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:16.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.719+0000 7f085891b640 4 set_mon_vals no callback set 2024-03-25T18:18:16.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.719+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f084c01ad60 con 0x7f08541430c0 2024-03-25T18:18:16.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.720+0000 7f085a11e640 10 monclient: get_auth_request con 0x7f083806df70 auth_method 0 2024-03-25T18:18:16.721 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.720+0000 7f084affd640 10 client.4402.objecter ms_handle_connect 0x7f083806df70 2024-03-25T18:18:16.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.725+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f084c054a70 con 0x7f08541430c0 2024-03-25T18:18:16.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.725+0000 7f084affd640 10 client.4402.objecter ms_dispatch 0x7f0854000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:16.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.725+0000 7f084affd640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:16.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.725+0000 7f084affd640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:16.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.725+0000 7f084affd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T18:18:16.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.726+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f084c0503b0 con 0x7f08541430c0 2024-03-25T18:18:16.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.726+0000 7f084affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:16.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.726+0000 7f084affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:16.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.939+0000 7f085c382640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:16.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.939+0000 7f085c382640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:16.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.939+0000 7f085c382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:16.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.939+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08540b3e60 con 0x7f08541430c0 2024-03-25T18:18:16.994 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.993+0000 7f084affd640 1 -- 172.21.15.162:0/370982945 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f084c0524b0 con 0x7f08541430c0 2024-03-25T18:18:16.994 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.993+0000 7f084affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:16.994 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.993+0000 7f084affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:16.995 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:18:16.995 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:16.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 10 librados: watch_flush enter 2024-03-25T18:18:16.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 10 librados: watch_flush exit 2024-03-25T18:18:16.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T18:18:16.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 10 client.4402.objecter successfully canceled tick 2024-03-25T18:18:16.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 >> v1:172.21.15.191:6800/1552703795 conn(0x7f083806df70 legacy=0x7f0838070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 10 monclient: shutdown 2024-03-25T18:18:16.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:16.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 >> v1:172.21.15.191:6789/0 conn(0x7f08541430c0 legacy=0x7f0854138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 shutdown_connections 2024-03-25T18:18:16.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 >> 172.21.15.162:0/370982945 conn(0x7f08540b7ea0 msgr2=0x7f085413d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:16.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 shutdown_connections 2024-03-25T18:18:16.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 1 -- 172.21.15.162:0/370982945 wait complete. 2024-03-25T18:18:16.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:16.996+0000 7f085c382640 1 librados: shutdown 2024-03-25T18:18:17.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:22.611 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:04:34.839 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.662 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:04:34.890 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.702 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:04:34.930 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.726 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:04:34.942 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.729 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:04:34.946 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.731 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:04:34.959 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.739 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:04:34.966 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.766 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:04:34.982 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:22.779 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:04:35.007 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:23.816 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.926+0000 7f63b8c25640 10 monclient: get_monmap_and_config 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.926+0000 7f63b8c25640 10 monclient: build_initial_monmap 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.926+0000 7f63b8c25640 10 monclient: monmap: 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:18:23.927393+0000 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:18:23.927393+0000 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:23.927 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:23.928 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:23.928 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:18:23.928 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:18:23.928 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.927+0000 7f63b8c25640 1 Processor -- start 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.927+0000 7f63b8c25640 1 -- start start 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.927+0000 7f63b8c25640 10 monclient: init 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.927+0000 7f63b8c25640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.927+0000 7f63b8c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.927+0000 7f63b8c25640 10 monclient(hunting): picked mon.noname-a con 0x7f63b40a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.927+0000 7f63b8c25640 10 monclient(hunting): picked mon.noname-c con 0x7f63b413f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.928+0000 7f63b8c25640 10 monclient(hunting): picked mon.noname-b con 0x7f63b41430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.928+0000 7f63b8c25640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b4142520 con 0x7f63b40a98f0 2024-03-25T18:18:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.928+0000 7f63b8c25640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40b28a0 con 0x7f63b413f710 2024-03-25T18:18:23.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.928+0000 7f63b8c25640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40b2a30 con 0x7f63b41430c0 2024-03-25T18:18:23.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.928+0000 7f63b8c25640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:23.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.928+0000 7f63b8c25640 10 monclient(hunting): authenticate will time out at 7803.201714s 2024-03-25T18:18:23.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.938+0000 7f63b259c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f63b40a98f0 0x7f63b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47016/0 (socket says 172.21.15.162:47016) 2024-03-25T18:18:23.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.938+0000 7f63b259c640 1 -- 172.21.15.162:0/4010591386 learned_addr learned my addr 172.21.15.162:0/4010591386 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:23.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.950+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751952463 0 0) 0x7f63b4142520 con 0x7f63b40a98f0 2024-03-25T18:18:23.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.950+0000 7f63b159a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:23.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.950+0000 7f63b159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:23.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.950+0000 7f63b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:23.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.950+0000 7f63b159a640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:18:23.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.950+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f639c003520 con 0x7f63b40a98f0 2024-03-25T18:18:23.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.954+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576393131 0 0) 0x7f63b40b28a0 con 0x7f63b413f710 2024-03-25T18:18:23.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.954+0000 7f63b159a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:23.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.954+0000 7f63b159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:23.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.954+0000 7f63b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:23.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.954+0000 7f63b159a640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:18:23.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.954+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4142520 con 0x7f63b413f710 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 749309818 0 0) 0x7f639c003520 con 0x7f63b40a98f0 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 >> v1:172.21.15.162:6790/0 conn(0x7f63b41430c0 legacy=0x7f63b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 >> v1:172.21.15.191:6789/0 conn(0x7f63b413f710 legacy=0x7f63b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 10 monclient: _finish_hunting 0 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 1 monclient: found mon.noname-a 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40b44c0 con 0x7f63b40a98f0 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 10 monclient: _finish_auth 0 2024-03-25T18:18:23.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 10 monclient: _check_auth_tickets 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f63a4001690 con 0x7f63b40a98f0 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b8c25640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b8c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b159a640 10 monclient: dump: 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:23.961 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:23.962 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:23.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.959+0000 7f63b8c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.971+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a40017f0 con 0x7f63b40a98f0 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.971+0000 7f63b159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.971+0000 7f63b159a640 1 -- 172.21.15.162:0/4010591386 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f63a4001c00 con 0x7f63b40a98f0 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.971+0000 7f63b8c25640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.971+0000 7f63b8c25640 4 set_mon_vals no callback set 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.971+0000 7f63b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.971+0000 7f63b159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.971+0000 7f63b159a640 10 monclient: dump: 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:23.972 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:23.973 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.973 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:23.973 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:23.973 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:23.973 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:23.973 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:23.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.972+0000 7f63b8c25640 10 monclient: shutdown 2024-03-25T18:18:23.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.972+0000 7f63b8c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:23.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.972+0000 7f63b8c25640 1 -- 172.21.15.162:0/4010591386 >> v1:172.21.15.162:6789/0 conn(0x7f63b40a98f0 legacy=0x7f63b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.972+0000 7f63b8c25640 1 -- 172.21.15.162:0/4010591386 shutdown_connections 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.972+0000 7f63b8c25640 1 -- 172.21.15.162:0/4010591386 >> 172.21.15.162:0/4010591386 conn(0x7f63b40b7ea0 msgr2=0x7f63b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.972+0000 7f63b8c25640 1 -- 172.21.15.162:0/4010591386 shutdown_connections 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.972+0000 7f63b8c25640 1 -- 172.21.15.162:0/4010591386 wait complete. 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 10 monclient: build_initial_monmap 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 10 monclient: monmap: 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:23.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 1 librados: starting msgr at 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 1 librados: starting objecter 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 1 Processor -- start 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 1 -- start start 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 1 librados: setting wanted keys 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 1 librados: calling monclient init 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.973+0000 7f63b8c25640 10 monclient: init 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 10 monclient(hunting): picked mon.noname-a con 0x7f63b40a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 10 monclient(hunting): picked mon.noname-b con 0x7f63b413f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 10 monclient(hunting): picked mon.noname-c con 0x7f63b41430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40b7ce0 con 0x7f63b40a98f0 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b41319b0 con 0x7f63b413f710 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b40b7800 con 0x7f63b41430c0 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:23.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.974+0000 7f63b8c25640 10 monclient(hunting): authenticate will time out at 7803.248132s 2024-03-25T18:18:23.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.976+0000 7f63b259c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f63b40a98f0 0x7f63b41dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:47054/0 (socket says 172.21.15.162:47054) 2024-03-25T18:18:23.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.976+0000 7f63b259c640 1 -- 172.21.15.162:0/261816096 learned_addr learned my addr 172.21.15.162:0/261816096 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:23.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.978+0000 7f63a2ffd640 10 client.?.objecter ms_handle_connect 0x7f63b40a98f0 2024-03-25T18:18:23.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.978+0000 7f63a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:23.983 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.982+0000 7f63a2ffd640 10 client.?.objecter ms_handle_connect 0x7f63b41430c0 2024-03-25T18:18:23.983 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.982+0000 7f63a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:23.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.985+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918458728 0 0) 0x7f63b40b7ce0 con 0x7f63b40a98f0 2024-03-25T18:18:23.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.985+0000 7f63a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:23.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.985+0000 7f63a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:23.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.985+0000 7f63a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:23.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.985+0000 7f63a2ffd640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:18:23.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.985+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6394004eb0 con 0x7f63b40a98f0 2024-03-25T18:18:23.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.995+0000 7f63a2ffd640 10 client.?.objecter ms_handle_connect 0x7f63b413f710 2024-03-25T18:18:23.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.995+0000 7f63a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:24.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.999+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127877604 0 0) 0x7f63b40b7800 con 0x7f63b41430c0 2024-03-25T18:18:24.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.999+0000 7f63a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:24.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.999+0000 7f63a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:24.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.999+0000 7f63a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:24.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.999+0000 7f63a2ffd640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:18:24.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:23.999+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b40b7ce0 con 0x7f63b41430c0 2024-03-25T18:18:24.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.000+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494211490 0 0) 0x7f63b41319b0 con 0x7f63b413f710 2024-03-25T18:18:24.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.000+0000 7f63a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:24.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.000+0000 7f63a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:24.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.000+0000 7f63a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:24.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.000+0000 7f63a2ffd640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:18:24.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.000+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b40b7800 con 0x7f63b413f710 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.005+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1011872895 0 0) 0x7f63b40b7800 con 0x7f63b413f710 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.006+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63b41319b0 con 0x7f63b413f710 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.006+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f63a8001930 con 0x7f63b413f710 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.006+0000 7f63a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.006+0000 7f63a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.006+0000 7f63a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:24.007 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:24.008 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:24.008 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:24.008 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:24.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.006+0000 7f63a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.010+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3317263416 0 0) 0x7f6394004eb0 con 0x7f63b40a98f0 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.010+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63b40b7800 con 0x7f63b40a98f0 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.010+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f63a4001c40 con 0x7f63b40a98f0 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.010+0000 7f63a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.010+0000 7f63a2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.010+0000 7f63a2ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:24.011 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:24.012 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:24.012 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:24.012 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:24.012 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:24.012 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:24.012 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:24.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1333645752 0 0) 0x7f63b41319b0 con 0x7f63b413f710 2024-03-25T18:18:24.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 >> v1:172.21.15.162:6790/0 conn(0x7f63b41430c0 legacy=0x7f63b41312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 >> v1:172.21.15.162:6789/0 conn(0x7f63b40a98f0 legacy=0x7f63b41dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 1 monclient: found mon.b 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4136f50 con 0x7f63b413f710 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 5 monclient: authenticate success, global_id 4408 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 10 monclient: _renew_subs 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b41350f0 con 0x7f63b413f710 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 10 monclient: _renew_subs 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b4135c60 con 0x7f63b413f710 2024-03-25T18:18:24.017 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.015+0000 7f63b8c25640 1 librados: init done 2024-03-25T18:18:24.018 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.016+0000 7f63b8c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:24.018 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.016+0000 7f63b8c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:24.018 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.016+0000 7f63b8c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:24.018 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.016+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b40627b0 con 0x7f63b413f710 2024-03-25T18:18:24.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.302+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a8003610 con 0x7f63b413f710 2024-03-25T18:18:24.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.302+0000 7f63a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.302+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f63a8001d80 con 0x7f63b413f710 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.302+0000 7f63a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.302+0000 7f63a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.302+0000 7f63a2ffd640 10 monclient: dump: 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:24.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.302+0000 7f63b0d99640 4 set_mon_vals no callback set 2024-03-25T18:18:24.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.303+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f63a80197d0 con 0x7f63b413f710 2024-03-25T18:18:24.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.304+0000 7f63b259c640 10 monclient: get_auth_request con 0x7f639406df70 auth_method 0 2024-03-25T18:18:24.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.304+0000 7f63a2ffd640 10 client.4408.objecter ms_handle_connect 0x7f639406df70 2024-03-25T18:18:24.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.311+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f63a8053510 con 0x7f63b413f710 2024-03-25T18:18:24.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.311+0000 7f63a2ffd640 10 client.4408.objecter ms_dispatch 0x7f63b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:24.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.311+0000 7f63a2ffd640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:24.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.311+0000 7f63a2ffd640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:24.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.311+0000 7f63a2ffd640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T18:18:24.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.312+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63a804ee50 con 0x7f63b413f710 2024-03-25T18:18:24.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.312+0000 7f63a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:24.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.312+0000 7f63a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:24.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.562+0000 7f63b8c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:24.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.562+0000 7f63b8c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:24.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.562+0000 7f63b8c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:24.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.562+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63b413e270 con 0x7f63b413f710 2024-03-25T18:18:24.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.598+0000 7f63a2ffd640 1 -- 172.21.15.162:0/261816096 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f63a8050f50 con 0x7f63b413f710 2024-03-25T18:18:24.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.598+0000 7f63a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:24.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.598+0000 7f63a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:24.599 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:18:24.599 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:24.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 10 librados: watch_flush enter 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 10 librados: watch_flush exit 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 10 client.4408.objecter successfully canceled tick 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 >> v1:172.21.15.191:6800/1552703795 conn(0x7f639406df70 legacy=0x7f6394070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 10 monclient: shutdown 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 >> v1:172.21.15.191:6789/0 conn(0x7f63b413f710 legacy=0x7f63b4130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 shutdown_connections 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 >> 172.21.15.162:0/261816096 conn(0x7f63b40b7ea0 msgr2=0x7f63b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 shutdown_connections 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 1 -- 172.21.15.162:0/261816096 wait complete. 2024-03-25T18:18:24.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:24.600+0000 7f63b8c25640 1 librados: shutdown 2024-03-25T18:18:24.614 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:31.420 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:31.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.537+0000 7f32ba55c640 10 monclient: get_monmap_and_config 2024-03-25T18:18:31.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.537+0000 7f32ba55c640 10 monclient: build_initial_monmap 2024-03-25T18:18:31.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.537+0000 7f32ba55c640 10 monclient: monmap: 2024-03-25T18:18:31.538 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:31.538 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:31.539 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:18:31.538660+0000 2024-03-25T18:18:31.539 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:18:31.538660+0000 2024-03-25T18:18:31.539 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:31.539 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:31.539 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:31.539 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:18:31.539 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:18:31.539 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:31.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.538+0000 7f32ba55c640 1 Processor -- start 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.538+0000 7f32ba55c640 1 -- start start 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.538+0000 7f32ba55c640 10 monclient: init 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.539+0000 7f32ba55c640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.539+0000 7f32ba55c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.539+0000 7f32ba55c640 10 monclient(hunting): picked mon.noname-c con 0x7f32b40b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.539+0000 7f32ba55c640 10 monclient(hunting): picked mon.noname-b con 0x7f32b40b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.539+0000 7f32ba55c640 10 monclient(hunting): picked mon.noname-a con 0x7f32b414be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.539+0000 7f32ba55c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40af940 con 0x7f32b414be90 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.539+0000 7f32ba55c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40b0a70 con 0x7f32b40b4d60 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.539+0000 7f32ba55c640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40b0d50 con 0x7f32b40b1730 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.540+0000 7f32ba55c640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:31.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.540+0000 7f32ba55c640 10 monclient(hunting): authenticate will time out at 7810.813713s 2024-03-25T18:18:31.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.556+0000 7f32b3fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f32b40b4d60 0x7f32b413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:44928/0 (socket says 172.21.15.162:44928) 2024-03-25T18:18:31.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.556+0000 7f32b3fff640 1 -- 172.21.15.162:0/765855619 learned_addr learned my addr 172.21.15.162:0/765855619 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:31.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.590+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188097435 0 0) 0x7f32b40af940 con 0x7f32b414be90 2024-03-25T18:18:31.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.590+0000 7f32b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:31.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.590+0000 7f32b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:31.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.590+0000 7f32b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:31.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.590+0000 7f32b2ffd640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:18:31.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.590+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f329c003520 con 0x7f32b414be90 2024-03-25T18:18:31.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.656+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60428923 0 0) 0x7f32b40b0a70 con 0x7f32b40b4d60 2024-03-25T18:18:31.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.657+0000 7f32b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:31.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.657+0000 7f32b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:31.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.657+0000 7f32b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:31.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.657+0000 7f32b2ffd640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:18:31.658 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.657+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b40af940 con 0x7f32b40b4d60 2024-03-25T18:18:31.662 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.661+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638530487 0 0) 0x7f32b40b0d50 con 0x7f32b40b1730 2024-03-25T18:18:31.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.661+0000 7f32b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:31.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.661+0000 7f32b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:31.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.661+0000 7f32b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:31.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.661+0000 7f32b2ffd640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:18:31.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.661+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b40b0a70 con 0x7f32b40b1730 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 718255195 0 0) 0x7f32b40af940 con 0x7f32b40b4d60 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 >> v1:172.21.15.162:6790/0 conn(0x7f32b40b1730 legacy=0x7f32b40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 >> v1:172.21.15.162:6789/0 conn(0x7f32b414be90 legacy=0x7f32b414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b40b1030 con 0x7f32b40b4d60 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f32a4001670 con 0x7f32b40b4d60 2024-03-25T18:18:31.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32ba55c640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32ba55c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32b2ffd640 10 monclient: dump: 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:31.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.671+0000 7f32ba55c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:31.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.683+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a40017d0 con 0x7f32b40b4d60 2024-03-25T18:18:31.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.683+0000 7f32b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.683+0000 7f32b2ffd640 1 -- 172.21.15.162:0/765855619 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f32a4001be0 con 0x7f32b40b4d60 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.683+0000 7f32b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.683+0000 7f32b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.683+0000 7f32b2ffd640 10 monclient: dump: 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:31.685 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:31.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.683+0000 7f32ba55c640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:31.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.683+0000 7f32ba55c640 4 set_mon_vals no callback set 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.684+0000 7f32ba55c640 10 monclient: shutdown 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.684+0000 7f32ba55c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.684+0000 7f32ba55c640 1 -- 172.21.15.162:0/765855619 >> v1:172.21.15.191:6789/0 conn(0x7f32b40b4d60 legacy=0x7f32b413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.684+0000 7f32ba55c640 1 -- 172.21.15.162:0/765855619 shutdown_connections 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.684+0000 7f32ba55c640 1 -- 172.21.15.162:0/765855619 >> 172.21.15.162:0/765855619 conn(0x7f32b40617a0 msgr2=0x7f32b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.684+0000 7f32ba55c640 1 -- 172.21.15.162:0/765855619 shutdown_connections 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.684+0000 7f32ba55c640 1 -- 172.21.15.162:0/765855619 wait complete. 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.685+0000 7f32ba55c640 10 monclient: build_initial_monmap 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.685+0000 7f32ba55c640 10 monclient: monmap: 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:31.687 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.685+0000 7f32ba55c640 1 librados: starting msgr at 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.685+0000 7f32ba55c640 1 librados: starting objecter 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.685+0000 7f32ba55c640 1 Processor -- start 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 1 -- start start 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 1 librados: setting wanted keys 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 1 librados: calling monclient init 2024-03-25T18:18:31.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 10 monclient: init 2024-03-25T18:18:31.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:31.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:31.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 10 monclient(hunting): picked mon.noname-b con 0x7f32b40b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:31.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 10 monclient(hunting): picked mon.noname-c con 0x7f32b40b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:31.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.686+0000 7f32ba55c640 10 monclient(hunting): picked mon.noname-a con 0x7f32b414be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:31.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.687+0000 7f32ba55c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b41d7ad0 con 0x7f32b414be90 2024-03-25T18:18:31.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.687+0000 7f32ba55c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40aac20 con 0x7f32b40b1730 2024-03-25T18:18:31.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.687+0000 7f32ba55c640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b41d8670 con 0x7f32b40b4d60 2024-03-25T18:18:31.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.687+0000 7f32ba55c640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:31.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.687+0000 7f32ba55c640 10 monclient(hunting): authenticate will time out at 7810.960717s 2024-03-25T18:18:31.699 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.698+0000 7f32b3fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f32b40b1730 0x7f32b40a9470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:45018/0 (socket says 172.21.15.162:45018) 2024-03-25T18:18:31.699 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.698+0000 7f32b3fff640 1 -- 172.21.15.162:0/2514482381 learned_addr learned my addr 172.21.15.162:0/2514482381 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:31.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.700+0000 7f32b0ff9640 10 client.?.objecter ms_handle_connect 0x7f32b40b1730 2024-03-25T18:18:31.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.700+0000 7f32b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:31.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.710+0000 7f32b0ff9640 10 client.?.objecter ms_handle_connect 0x7f32b40b4d60 2024-03-25T18:18:31.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.710+0000 7f32b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:31.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.712+0000 7f32b0ff9640 10 client.?.objecter ms_handle_connect 0x7f32b414be90 2024-03-25T18:18:31.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.712+0000 7f32b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:31.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.714+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437669887 0 0) 0x7f32b40aac20 con 0x7f32b40b1730 2024-03-25T18:18:31.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.714+0000 7f32b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:31.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.714+0000 7f32b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:31.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.714+0000 7f32b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:31.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.714+0000 7f32b0ff9640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:18:31.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.714+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a0004eb0 con 0x7f32b40b1730 2024-03-25T18:18:31.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.719+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779500107 0 0) 0x7f32b41d8670 con 0x7f32b40b4d60 2024-03-25T18:18:31.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.719+0000 7f32b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:31.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.719+0000 7f32b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:31.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.719+0000 7f32b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:31.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.719+0000 7f32b0ff9640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:18:31.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.719+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b40aac20 con 0x7f32b40b4d60 2024-03-25T18:18:31.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.722+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651127752 0 0) 0x7f32b41d7ad0 con 0x7f32b414be90 2024-03-25T18:18:31.724 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.722+0000 7f32b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:31.724 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.722+0000 7f32b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:31.724 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.722+0000 7f32b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:31.724 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.722+0000 7f32b0ff9640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:18:31.724 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.722+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32b41d8670 con 0x7f32b414be90 2024-03-25T18:18:31.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.737+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813848679 0 0) 0x7f32a0004eb0 con 0x7f32b40b1730 2024-03-25T18:18:31.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.737+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32b41d7ad0 con 0x7f32b40b1730 2024-03-25T18:18:31.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.737+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f32a4001c20 con 0x7f32b40b1730 2024-03-25T18:18:31.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.737+0000 7f32b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:31.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.737+0000 7f32b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:31.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.737+0000 7f32b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:31.738 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:31.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.737+0000 7f32b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.747+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2877052234 0 0) 0x7f32b40aac20 con 0x7f32b40b4d60 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.748+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32a0004eb0 con 0x7f32b40b4d60 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.748+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f32a8001870 con 0x7f32b40b4d60 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.748+0000 7f32b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.748+0000 7f32b0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.748+0000 7f32b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:31.749 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:31.750 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:31.750 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:31.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2522390226 0 0) 0x7f32b41d7ad0 con 0x7f32b40b1730 2024-03-25T18:18:31.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 >> v1:172.21.15.162:6790/0 conn(0x7f32b40b4d60 legacy=0x7f32b41d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 >> v1:172.21.15.162:6789/0 conn(0x7f32b414be90 legacy=0x7f32b41e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:31.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 1 monclient: found mon.b 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b41d8800 con 0x7f32b40b1730 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 10 monclient: _renew_subs 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b4140380 con 0x7f32b40b1730 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 10 monclient: _renew_subs 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:31.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b4143320 con 0x7f32b40b1730 2024-03-25T18:18:31.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.756+0000 7f32ba55c640 1 librados: init done 2024-03-25T18:18:31.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.757+0000 7f32ba55c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:31.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.757+0000 7f32ba55c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:31.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.757+0000 7f32ba55c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:31.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:31.757+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b41d78a0 con 0x7f32b40b1730 2024-03-25T18:18:32.050 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.049+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a4004a90 con 0x7f32b40b1730 2024-03-25T18:18:32.050 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.049+0000 7f32b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:32.050 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.049+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f32a4004c40 con 0x7f32b40b1730 2024-03-25T18:18:32.050 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.049+0000 7f32b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:32.050 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.049+0000 7f32b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:32.050 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.049+0000 7f32b0ff9640 10 monclient: dump: 2024-03-25T18:18:32.050 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:32.050 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:32.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.049+0000 7f32b27fc640 4 set_mon_vals no callback set 2024-03-25T18:18:32.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.049+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f32a4004e60 con 0x7f32b40b1730 2024-03-25T18:18:32.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.050+0000 7f32b37fe640 10 monclient: get_auth_request con 0x7f32a006df50 auth_method 0 2024-03-25T18:18:32.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.050+0000 7f32b0ff9640 10 client.4414.objecter ms_handle_connect 0x7f32a006df50 2024-03-25T18:18:32.055 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.054+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f32a40547f0 con 0x7f32b40b1730 2024-03-25T18:18:32.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.054+0000 7f32b0ff9640 10 client.4414.objecter ms_dispatch 0x7f32b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:32.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.054+0000 7f32b0ff9640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:32.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.054+0000 7f32b0ff9640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:32.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.054+0000 7f32b0ff9640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T18:18:32.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.055+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32a4050300 con 0x7f32b40b1730 2024-03-25T18:18:32.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.055+0000 7f32b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:32.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.055+0000 7f32b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:32.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.304+0000 7f32ba55c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:32.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.304+0000 7f32ba55c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:32.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.304+0000 7f32ba55c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:32.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.304+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32b40627b0 con 0x7f32b40b1730 2024-03-25T18:18:32.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.345+0000 7f32b0ff9640 1 -- 172.21.15.162:0/2514482381 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f32a4052230 con 0x7f32b40b1730 2024-03-25T18:18:32.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.345+0000 7f32b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:32.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.345+0000 7f32b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:32.346 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:18:32.347 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:32.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.347+0000 7f32ba55c640 10 librados: watch_flush enter 2024-03-25T18:18:32.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.347+0000 7f32ba55c640 10 librados: watch_flush exit 2024-03-25T18:18:32.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.347+0000 7f32ba55c640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.347+0000 7f32ba55c640 10 client.4414.objecter successfully canceled tick 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 >> v1:172.21.15.191:6800/1552703795 conn(0x7f32a006df50 legacy=0x7f32a0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 10 monclient: shutdown 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 >> v1:172.21.15.191:6789/0 conn(0x7f32b40b1730 legacy=0x7f32b40a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 shutdown_connections 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 >> 172.21.15.162:0/2514482381 conn(0x7f32b40617a0 msgr2=0x7f32b4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 shutdown_connections 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 1 -- 172.21.15.162:0/2514482381 wait complete. 2024-03-25T18:18:32.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:32.348+0000 7f32ba55c640 1 librados: shutdown 2024-03-25T18:18:32.361 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:39.167 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.280+0000 7f4db42c8640 10 monclient: get_monmap_and_config 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.280+0000 7f4db42c8640 10 monclient: build_initial_monmap 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.280+0000 7f4db42c8640 10 monclient: monmap: 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:18:39.281967+0000 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:18:39.281967+0000 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:18:39.282 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:39.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.281+0000 7f4db42c8640 1 Processor -- start 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.281+0000 7f4db42c8640 1 -- start start 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.281+0000 7f4db42c8640 10 monclient: init 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 10 monclient(hunting): picked mon.noname-c con 0x7f4dac0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 10 monclient(hunting): picked mon.noname-a con 0x7f4dac0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 10 monclient(hunting): picked mon.noname-b con 0x7f4dac14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac0af940 con 0x7f4dac0b1730 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac0b0a70 con 0x7f4dac0b4d60 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac0b0d50 con 0x7f4dac14be90 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:39.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.282+0000 7f4db42c8640 10 monclient(hunting): authenticate will time out at 7818.556454s 2024-03-25T18:18:39.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.291+0000 7f4db2064640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4dac0b4d60 0x7f4dac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:39164/0 (socket says 172.21.15.162:39164) 2024-03-25T18:18:39.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.291+0000 7f4db2064640 1 -- 172.21.15.162:0/113092203 learned_addr learned my addr 172.21.15.162:0/113092203 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:39.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.297+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28470653 0 0) 0x7f4dac0b0a70 con 0x7f4dac0b4d60 2024-03-25T18:18:39.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.297+0000 7f4db1062640 10 monclient(hunting): _negotiate 2024-03-25T18:18:39.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.297+0000 7f4db1062640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:39.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.297+0000 7f4db1062640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:39.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.297+0000 7f4db1062640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:18:39.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.297+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d98003520 con 0x7f4dac0b4d60 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 67539751 0 0) 0x7f4d98003520 con 0x7f4dac0b4d60 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 >> v1:172.21.15.162:6790/0 conn(0x7f4dac14be90 legacy=0x7f4dac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 >> v1:172.21.15.162:6789/0 conn(0x7f4dac0b1730 legacy=0x7f4dac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 10 monclient: _finish_hunting 0 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 1 monclient: found mon.noname-c 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac0b1030 con 0x7f4dac0b4d60 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 10 monclient: _finish_auth 0 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 10 monclient: _check_auth_tickets 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4da8001690 con 0x7f4dac0b4d60 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.312+0000 7f4db1062640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:39.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.313+0000 7f4db1062640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.313+0000 7f4db1062640 10 monclient: dump: 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.313+0000 7f4db42c8640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:18:39.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.313+0000 7f4db42c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.316+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4da80017f0 con 0x7f4dac0b4d60 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.316+0000 7f4db1062640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.316+0000 7f4db1062640 1 -- 172.21.15.162:0/113092203 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4da8001c40 con 0x7f4dac0b4d60 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.316+0000 7f4db42c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.316+0000 7f4db42c8640 4 set_mon_vals no callback set 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.316+0000 7f4db1062640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.316+0000 7f4db1062640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.316+0000 7f4db1062640 10 monclient: dump: 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:39.318 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:39.319 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:39.319 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:39.319 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:39.319 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.317+0000 7f4db42c8640 10 monclient: shutdown 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.317+0000 7f4db42c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.317+0000 7f4db42c8640 1 -- 172.21.15.162:0/113092203 >> v1:172.21.15.191:6789/0 conn(0x7f4dac0b4d60 legacy=0x7f4dac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.318+0000 7f4db42c8640 1 -- 172.21.15.162:0/113092203 shutdown_connections 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.318+0000 7f4db42c8640 1 -- 172.21.15.162:0/113092203 >> 172.21.15.162:0/113092203 conn(0x7f4dac0617a0 msgr2=0x7f4dac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.318+0000 7f4db42c8640 1 -- 172.21.15.162:0/113092203 shutdown_connections 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.318+0000 7f4db42c8640 1 -- 172.21.15.162:0/113092203 wait complete. 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.318+0000 7f4db42c8640 10 monclient: build_initial_monmap 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.318+0000 7f4db42c8640 10 monclient: monmap: 2024-03-25T18:18:39.320 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.319+0000 7f4db42c8640 1 librados: starting msgr at 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.319+0000 7f4db42c8640 1 librados: starting objecter 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.319+0000 7f4db42c8640 1 Processor -- start 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.319+0000 7f4db42c8640 1 -- start start 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.319+0000 7f4db42c8640 1 librados: setting wanted keys 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.319+0000 7f4db42c8640 1 librados: calling monclient init 2024-03-25T18:18:39.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.319+0000 7f4db42c8640 10 monclient: init 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 10 monclient(hunting): picked mon.noname-c con 0x7f4dac0b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 10 monclient(hunting): picked mon.noname-b con 0x7f4dac0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 10 monclient(hunting): picked mon.noname-a con 0x7f4dac14be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac136110 con 0x7f4dac14be90 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac138010 con 0x7f4dac0b4d60 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dac1382f0 con 0x7f4dac0b1730 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:39.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.320+0000 7f4db42c8640 10 monclient(hunting): authenticate will time out at 7818.593952s 2024-03-25T18:18:39.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.327+0000 7f4db1863640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4dac0b4d60 0x7f4dac1dc3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:39210/0 (socket says 172.21.15.162:39210) 2024-03-25T18:18:39.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.327+0000 7f4db1863640 1 -- 172.21.15.162:0/4143586634 learned_addr learned my addr 172.21.15.162:0/4143586634 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:39.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.327+0000 7f4da2ffd640 10 client.?.objecter ms_handle_connect 0x7f4dac0b4d60 2024-03-25T18:18:39.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.327+0000 7f4da2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:39.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.333+0000 7f4da2ffd640 10 client.?.objecter ms_handle_connect 0x7f4dac14be90 2024-03-25T18:18:39.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.333+0000 7f4da2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:39.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.334+0000 7f4da2ffd640 10 client.?.objecter ms_handle_connect 0x7f4dac0b1730 2024-03-25T18:18:39.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.334+0000 7f4da2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:39.340 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.339+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328898672 0 0) 0x7f4dac1382f0 con 0x7f4dac0b1730 2024-03-25T18:18:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.339+0000 7f4da2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.339+0000 7f4da2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.339+0000 7f4da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.339+0000 7f4da2ffd640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:18:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.339+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d90004eb0 con 0x7f4dac0b1730 2024-03-25T18:18:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.340+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960492115 0 0) 0x7f4dac138010 con 0x7f4dac0b4d60 2024-03-25T18:18:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.340+0000 7f4da2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.340+0000 7f4da2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.340+0000 7f4da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.340+0000 7f4da2ffd640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:18:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.340+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dac1382f0 con 0x7f4dac0b4d60 2024-03-25T18:18:39.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.343+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667470171 0 0) 0x7f4dac136110 con 0x7f4dac14be90 2024-03-25T18:18:39.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.343+0000 7f4da2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:39.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.343+0000 7f4da2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:39.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.343+0000 7f4da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:39.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.343+0000 7f4da2ffd640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:18:39.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.343+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dac138010 con 0x7f4dac14be90 2024-03-25T18:18:39.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.348+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 649896778 0 0) 0x7f4d90004eb0 con 0x7f4dac0b1730 2024-03-25T18:18:39.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.348+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4dac136110 con 0x7f4dac0b1730 2024-03-25T18:18:39.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.348+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4da8001b00 con 0x7f4dac0b1730 2024-03-25T18:18:39.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.348+0000 7f4da2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:39.349 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.348+0000 7f4da2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.348+0000 7f4da2ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:39.350 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.348+0000 7f4da2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.358+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4055634179 0 0) 0x7f4dac1382f0 con 0x7f4dac0b4d60 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.358+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d90004eb0 con 0x7f4dac0b4d60 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.358+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4d9c0019b0 con 0x7f4dac0b4d60 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.358+0000 7f4da2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.358+0000 7f4da2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.358+0000 7f4da2ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:39.359 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:39.360 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:39.360 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:39.360 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2439940708 0 0) 0x7f4dac136110 con 0x7f4dac0b1730 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 >> v1:172.21.15.191:6789/0 conn(0x7f4dac0b4d60 legacy=0x7f4dac1dc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 >> v1:172.21.15.162:6789/0 conn(0x7f4dac14be90 legacy=0x7f4dac143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 1 monclient: found mon.c 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac1385d0 con 0x7f4dac0b1730 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4da2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4db42c8640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4db42c8640 10 monclient: _renew_subs 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4db42c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.364+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dac1dcb00 con 0x7f4dac0b1730 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 10 monclient: _renew_subs 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dac1442b0 con 0x7f4dac0b1730 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 1 librados: init done 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:18:39.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.365+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dac135f70 con 0x7f4dac0b1730 2024-03-25T18:18:39.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.650+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4da8001c60 con 0x7f4dac0b1730 2024-03-25T18:18:39.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.650+0000 7f4da2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:39.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.650+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4da8004bf0 con 0x7f4dac0b1730 2024-03-25T18:18:39.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.650+0000 7f4da2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:39.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.650+0000 7f4da2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:39.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.650+0000 7f4da2ffd640 10 monclient: dump: 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:39.652 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.650+0000 7f4db0861640 4 set_mon_vals no callback set 2024-03-25T18:18:39.653 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.651+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f4da801abc0 con 0x7f4dac0b1730 2024-03-25T18:18:39.653 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.651+0000 7f4db1863640 10 monclient: get_auth_request con 0x7f4d9006df30 auth_method 0 2024-03-25T18:18:39.653 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.652+0000 7f4da2ffd640 10 client.4424.objecter ms_handle_connect 0x7f4d9006df30 2024-03-25T18:18:39.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.658+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f4da80546c0 con 0x7f4dac0b1730 2024-03-25T18:18:39.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.658+0000 7f4da2ffd640 10 client.4424.objecter ms_dispatch 0x7f4dac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:39.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.658+0000 7f4da2ffd640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:39.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.658+0000 7f4da2ffd640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:39.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.658+0000 7f4da2ffd640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T18:18:39.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.658+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4da80501d0 con 0x7f4dac0b1730 2024-03-25T18:18:39.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.658+0000 7f4da2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:39.660 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.658+0000 7f4da2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:39.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.909+0000 7f4db42c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:39.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.909+0000 7f4db42c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:39.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.909+0000 7f4db42c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:18:39.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.909+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dac0b2270 con 0x7f4dac0b1730 2024-03-25T18:18:39.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.934+0000 7f4da2ffd640 1 -- 172.21.15.162:0/4143586634 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f4da8052100 con 0x7f4dac0b1730 2024-03-25T18:18:39.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.934+0000 7f4da2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:39.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.934+0000 7f4da2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:39.936 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:18:39.936 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:39.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 10 librados: watch_flush enter 2024-03-25T18:18:39.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 10 librados: watch_flush exit 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 10 client.4424.objecter successfully canceled tick 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 >> v1:172.21.15.191:6800/1552703795 conn(0x7f4d9006df30 legacy=0x7f4d900703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 10 monclient: shutdown 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 >> v1:172.21.15.162:6790/0 conn(0x7f4dac0b1730 legacy=0x7f4dac136d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 shutdown_connections 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 >> 172.21.15.162:0/4143586634 conn(0x7f4dac0617a0 msgr2=0x7f4dac0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 shutdown_connections 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 1 -- 172.21.15.162:0/4143586634 wait complete. 2024-03-25T18:18:39.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:39.937+0000 7f4db42c8640 1 librados: shutdown 2024-03-25T18:18:39.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:42.758 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:04:54.986 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.770 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:04:54.998 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.815 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:04:55.043 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.824 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:04:55.040 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.830 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:04:55.056 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.830 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:04:55.046 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.836 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:04:55.064 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.856 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:04:55.072 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:42.873 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:04:55.101 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:46.757 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:46.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.871+0000 7f1fddae6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.871+0000 7f1fddae6640 10 monclient: build_initial_monmap 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.871+0000 7f1fddae6640 10 monclient: monmap: 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:18:46.872089+0000 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:18:46.872089+0000 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:18:46.872 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:46.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.872+0000 7f1fddae6640 1 Processor -- start 2024-03-25T18:18:46.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.872+0000 7f1fddae6640 1 -- start start 2024-03-25T18:18:46.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.872+0000 7f1fddae6640 10 monclient: init 2024-03-25T18:18:46.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:46.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:46.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd813fc60 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:46.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd8070570 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:46.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd80727a0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:46.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd813c9f0 con 0x7f1fd80727a0 2024-03-25T18:18:46.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd806fac0 con 0x7f1fd8070570 2024-03-25T18:18:46.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd80715d0 con 0x7f1fd813fc60 2024-03-25T18:18:46.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:46.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.873+0000 7f1fddae6640 10 monclient(hunting): authenticate will time out at 7826.146926s 2024-03-25T18:18:46.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.883+0000 7f1fd6ffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1fd8070570 0x7f1fd8072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:36944/0 (socket says 172.21.15.162:36944) 2024-03-25T18:18:46.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.883+0000 7f1fd6ffd640 1 -- 172.21.15.162:0/1571744526 learned_addr learned my addr 172.21.15.162:0/1571744526 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:46.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.898+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382467869 0 0) 0x7f1fd806fac0 con 0x7f1fd8070570 2024-03-25T18:18:46.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:18:46.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc0005730 con 0x7f1fd8070570 2024-03-25T18:18:46.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390120553 0 0) 0x7f1fd80715d0 con 0x7f1fd813fc60 2024-03-25T18:18:46.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:18:46.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.899+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd806fac0 con 0x7f1fd813fc60 2024-03-25T18:18:46.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463345803 0 0) 0x7f1fc0005730 con 0x7f1fd8070570 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 >> v1:172.21.15.162:6790/0 conn(0x7f1fd813fc60 legacy=0x7f1fd806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 >> v1:172.21.15.162:6789/0 conn(0x7f1fd80727a0 legacy=0x7f1fd814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 1 monclient: found mon.noname-c 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd80718b0 con 0x7f1fd8070570 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fddae6640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:18:46.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.902+0000 7f1fddae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.907+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1fcc001650 con 0x7f1fd8070570 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.907+0000 7f1fd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.907+0000 7f1fd67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.907+0000 7f1fd67fc640 10 monclient: dump: 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.908 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:46.909 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:46.909 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:46.909 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:46.909 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:46.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.907+0000 7f1fddae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:46.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.914+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fcc0018a0 con 0x7f1fd8070570 2024-03-25T18:18:46.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.914+0000 7f1fd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.914+0000 7f1fd67fc640 1 -- 172.21.15.162:0/1571744526 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1fcc001c30 con 0x7f1fd8070570 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.914+0000 7f1fddae6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.914+0000 7f1fd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.914+0000 7f1fddae6640 4 set_mon_vals no callback set 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.914+0000 7f1fd67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.914+0000 7f1fd67fc640 10 monclient: dump: 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:46.916 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:46.917 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:46.917 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:46.917 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.915+0000 7f1fddae6640 10 monclient: shutdown 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.915+0000 7f1fddae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.915+0000 7f1fddae6640 1 -- 172.21.15.162:0/1571744526 >> v1:172.21.15.191:6789/0 conn(0x7f1fd8070570 legacy=0x7f1fd8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.915+0000 7f1fddae6640 1 -- 172.21.15.162:0/1571744526 shutdown_connections 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.915+0000 7f1fddae6640 1 -- 172.21.15.162:0/1571744526 >> 172.21.15.162:0/1571744526 conn(0x7f1fd80b4a80 msgr2=0x7f1fd80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.915+0000 7f1fddae6640 1 -- 172.21.15.162:0/1571744526 shutdown_connections 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.915+0000 7f1fddae6640 1 -- 172.21.15.162:0/1571744526 wait complete. 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.916+0000 7f1fddae6640 10 monclient: build_initial_monmap 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.916+0000 7f1fddae6640 10 monclient: monmap: 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:46.918 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.916+0000 7f1fddae6640 1 librados: starting msgr at 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.916+0000 7f1fddae6640 1 librados: starting objecter 2024-03-25T18:18:46.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.917+0000 7f1fddae6640 1 Processor -- start 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.917+0000 7f1fddae6640 1 -- start start 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.917+0000 7f1fddae6640 1 librados: setting wanted keys 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.917+0000 7f1fddae6640 1 librados: calling monclient init 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.917+0000 7f1fddae6640 10 monclient: init 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.918+0000 7f1fddae6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.918+0000 7f1fddae6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.918+0000 7f1fddae6640 10 monclient(hunting): picked mon.noname-c con 0x7f1fd8070570 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.918+0000 7f1fddae6640 10 monclient(hunting): picked mon.noname-a con 0x7f1fd80727a0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.918+0000 7f1fddae6640 10 monclient(hunting): picked mon.noname-b con 0x7f1fd813fc60 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.918+0000 7f1fddae6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd813a920 con 0x7f1fd80727a0 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.919+0000 7f1fddae6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd81335e0 con 0x7f1fd813fc60 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.919+0000 7f1fddae6640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd8143300 con 0x7f1fd8070570 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.919+0000 7f1fddae6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:46.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.919+0000 7f1fddae6640 10 monclient(hunting): authenticate will time out at 7826.192961s 2024-03-25T18:18:46.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.926+0000 7f1fd7fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1fd813fc60 0x7f1fd8132ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:36982/0 (socket says 172.21.15.162:36982) 2024-03-25T18:18:46.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.926+0000 7f1fd7fff640 1 -- 172.21.15.162:0/1379607814 learned_addr learned my addr 172.21.15.162:0/1379607814 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:46.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.928+0000 7f1fbffff640 10 client.?.objecter ms_handle_connect 0x7f1fd8070570 2024-03-25T18:18:46.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.929+0000 7f1fbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:46.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.929+0000 7f1fbffff640 10 client.?.objecter ms_handle_connect 0x7f1fd813fc60 2024-03-25T18:18:46.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.929+0000 7f1fbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:46.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223961211 0 0) 0x7f1fd8143300 con 0x7f1fd8070570 2024-03-25T18:18:46.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:18:46.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc4004eb0 con 0x7f1fd8070570 2024-03-25T18:18:46.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 client.?.objecter ms_handle_connect 0x7f1fd80727a0 2024-03-25T18:18:46.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:46.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074823601 0 0) 0x7f1fd81335e0 con 0x7f1fd813fc60 2024-03-25T18:18:46.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:18:46.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.939+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd8143300 con 0x7f1fd813fc60 2024-03-25T18:18:46.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.949+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384331953 0 0) 0x7f1fd813a920 con 0x7f1fd80727a0 2024-03-25T18:18:46.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.949+0000 7f1fbffff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.949+0000 7f1fbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.949+0000 7f1fbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.949+0000 7f1fbffff640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:18:46.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.949+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fd81335e0 con 0x7f1fd80727a0 2024-03-25T18:18:46.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.950+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3642003679 0 0) 0x7f1fd8143300 con 0x7f1fd813fc60 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.950+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fd813a920 con 0x7f1fd813fc60 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.950+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1fd0001bf0 con 0x7f1fd813fc60 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.950+0000 7f1fbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.950+0000 7f1fbffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.950+0000 7f1fbffff640 10 monclient(hunting): dump: 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:46.952 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:46.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.950+0000 7f1fbffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:46.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.951+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2130928335 0 0) 0x7f1fc4004eb0 con 0x7f1fd8070570 2024-03-25T18:18:46.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.951+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fd8143300 con 0x7f1fd8070570 2024-03-25T18:18:46.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.951+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1fc8001890 con 0x7f1fd8070570 2024-03-25T18:18:46.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.951+0000 7f1fbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.951+0000 7f1fbffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.951+0000 7f1fbffff640 10 monclient(hunting): dump: 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:46.954 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.957+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 208019392 0 0) 0x7f1fd81335e0 con 0x7f1fd80727a0 2024-03-25T18:18:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.957+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fc4004eb0 con 0x7f1fd80727a0 2024-03-25T18:18:46.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.960+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1fcc001bb0 con 0x7f1fd80727a0 2024-03-25T18:18:46.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.960+0000 7f1fbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.960+0000 7f1fbffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:46.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.960+0000 7f1fbffff640 10 monclient(hunting): dump: 2024-03-25T18:18:46.961 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:46.961 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:46.962 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.962 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:46.962 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.962 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:46.962 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:46.962 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:46.962 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:46.962 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:46.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2372152090 0 0) 0x7f1fd813a920 con 0x7f1fd813fc60 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 >> v1:172.21.15.162:6790/0 conn(0x7f1fd8070570 legacy=0x7f1fd813a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 >> v1:172.21.15.162:6789/0 conn(0x7f1fd80727a0 legacy=0x7f1fd81327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 1 monclient: found mon.b 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8139790 con 0x7f1fd813fc60 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 10 monclient: _finish_auth 0 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:18:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 10 monclient: _renew_subs 2024-03-25T18:18:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd81399e0 con 0x7f1fd813fc60 2024-03-25T18:18:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 10 monclient: _renew_subs 2024-03-25T18:18:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd8140530 con 0x7f1fd813fc60 2024-03-25T18:18:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.966+0000 7f1fddae6640 1 librados: init done 2024-03-25T18:18:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.967+0000 7f1fddae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.967+0000 7f1fddae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.967+0000 7f1fddae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:46.967+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd80627b0 con 0x7f1fd813fc60 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.260+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fd00020e0 con 0x7f1fd813fc60 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.260+0000 7f1fbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.260+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1fd0003b30 con 0x7f1fd813fc60 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.260+0000 7f1fbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.260+0000 7f1fbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.260+0000 7f1fbffff640 10 monclient: dump: 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:47.262 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:47.263 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:47.263 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:47.263 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:47.263 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:47.263 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:47.263 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:47.263 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.260+0000 7f1fd5ffb640 4 set_mon_vals no callback set 2024-03-25T18:18:47.263 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.261+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f1fd0003d10 con 0x7f1fd813fc60 2024-03-25T18:18:47.264 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.262+0000 7f1fd77fe640 10 monclient: get_auth_request con 0x7f1fc406dfc0 auth_method 0 2024-03-25T18:18:47.264 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.262+0000 7f1fbffff640 10 client.4426.objecter ms_handle_connect 0x7f1fc406dfc0 2024-03-25T18:18:47.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.267+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f1fd0054a60 con 0x7f1fd813fc60 2024-03-25T18:18:47.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.267+0000 7f1fbffff640 10 client.4426.objecter ms_dispatch 0x7f1fd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:47.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.267+0000 7f1fbffff640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:47.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.267+0000 7f1fbffff640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:47.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.267+0000 7f1fbffff640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T18:18:47.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.268+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fd00503a0 con 0x7f1fd813fc60 2024-03-25T18:18:47.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.268+0000 7f1fbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:47.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.268+0000 7f1fbffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:47.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.517+0000 7f1fddae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:47.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.517+0000 7f1fddae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:47.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.517+0000 7f1fddae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:47.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.517+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fd80af160 con 0x7f1fd813fc60 2024-03-25T18:18:47.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.538+0000 7f1fbffff640 1 -- 172.21.15.162:0/1379607814 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f1fd00524a0 con 0x7f1fd813fc60 2024-03-25T18:18:47.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.538+0000 7f1fbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:47.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.538+0000 7f1fbffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:47.540 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:18:47.540 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:47.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 10 librados: watch_flush enter 2024-03-25T18:18:47.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 10 librados: watch_flush exit 2024-03-25T18:18:47.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T18:18:47.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 10 client.4426.objecter successfully canceled tick 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 >> v1:172.21.15.191:6800/1552703795 conn(0x7f1fc406dfc0 legacy=0x7f1fc4070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 10 monclient: shutdown 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 >> v1:172.21.15.191:6789/0 conn(0x7f1fd813fc60 legacy=0x7f1fd8132ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 shutdown_connections 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.540+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 >> 172.21.15.162:0/1379607814 conn(0x7f1fd80b4a80 msgr2=0x7f1fd813ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.541+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 shutdown_connections 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.541+0000 7f1fddae6640 1 -- 172.21.15.162:0/1379607814 wait complete. 2024-03-25T18:18:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:47.541+0000 7f1fddae6640 1 librados: shutdown 2024-03-25T18:18:47.554 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:54.359 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:54.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.478+0000 7f3a137c6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:54.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.478+0000 7f3a137c6640 10 monclient: build_initial_monmap 2024-03-25T18:18:54.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.478+0000 7f3a137c6640 10 monclient: monmap: 2024-03-25T18:18:54.479 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:18:54.479547+0000 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:18:54.479547+0000 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:18:54.480 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:54.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.479+0000 7f3a137c6640 1 Processor -- start 2024-03-25T18:18:54.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.479+0000 7f3a137c6640 1 -- start start 2024-03-25T18:18:54.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.479+0000 7f3a137c6640 10 monclient: init 2024-03-25T18:18:54.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.479+0000 7f3a137c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:54.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.479+0000 7f3a137c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:54.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.480+0000 7f3a137c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0c0a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:54.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.480+0000 7f3a137c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0c13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:54.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.480+0000 7f3a137c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:54.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.480+0000 7f3a137c6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c142520 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.480+0000 7f3a137c6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0b28a0 con 0x7f3a0c13f710 2024-03-25T18:18:54.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.480+0000 7f3a137c6640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c0b2a30 con 0x7f3a0c1430c0 2024-03-25T18:18:54.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.480+0000 7f3a137c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:54.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.480+0000 7f3a137c6640 10 monclient(hunting): authenticate will time out at 7833.753852s 2024-03-25T18:18:54.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.490+0000 7f3a10d61640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3a0c13f710 0x7f3a0c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:59038/0 (socket says 172.21.15.162:59038) 2024-03-25T18:18:54.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.490+0000 7f3a10d61640 1 -- 172.21.15.162:0/999776155 learned_addr learned my addr 172.21.15.162:0/999776155 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:54.508 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.507+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310625707 0 0) 0x7f3a0c0b28a0 con 0x7f3a0c13f710 2024-03-25T18:18:54.508 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.507+0000 7f3a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:54.508 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.507+0000 7f3a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:54.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.507+0000 7f3a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:54.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.507+0000 7f3a03fff640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:18:54.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.507+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f8003520 con 0x7f3a0c13f710 2024-03-25T18:18:54.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.517+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374242840 0 0) 0x7f3a0c0b2a30 con 0x7f3a0c1430c0 2024-03-25T18:18:54.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.517+0000 7f3a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:54.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.517+0000 7f3a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:54.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.517+0000 7f3a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:54.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.517+0000 7f3a03fff640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:18:54.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.517+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a0c0b28a0 con 0x7f3a0c1430c0 2024-03-25T18:18:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.529+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723402124 0 0) 0x7f3a0c142520 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.529+0000 7f3a03fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.529+0000 7f3a03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:54.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.529+0000 7f3a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:54.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.529+0000 7f3a03fff640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:18:54.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.529+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a0c0b2a30 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114216109 0 0) 0x7f39f8003520 con 0x7f3a0c13f710 2024-03-25T18:18:54.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 >> v1:172.21.15.162:6790/0 conn(0x7f3a0c1430c0 legacy=0x7f3a0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 >> v1:172.21.15.162:6789/0 conn(0x7f3a0c0a98f0 legacy=0x7f3a0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:54.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 1 monclient: found mon.noname-c 2024-03-25T18:18:54.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:54.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:18:54.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c0b44c0 con 0x7f3a0c13f710 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f39fc001690 con 0x7f3a0c13f710 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a03fff640 10 monclient: dump: 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:54.536 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:54.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a137c6640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:18:54.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.534+0000 7f3a137c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:54.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.554+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39fc0017f0 con 0x7f3a0c13f710 2024-03-25T18:18:54.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.554+0000 7f3a03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:54.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.554+0000 7f3a03fff640 1 -- 172.21.15.162:0/999776155 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f39fc001c00 con 0x7f3a0c13f710 2024-03-25T18:18:54.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.554+0000 7f3a137c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:54.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.554+0000 7f3a137c6640 4 set_mon_vals no callback set 2024-03-25T18:18:54.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.554+0000 7f3a03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:54.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.554+0000 7f3a03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.554+0000 7f3a03fff640 10 monclient: dump: 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:54.556 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:54.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.555+0000 7f3a137c6640 10 monclient: shutdown 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.555+0000 7f3a137c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.555+0000 7f3a137c6640 1 -- 172.21.15.162:0/999776155 >> v1:172.21.15.191:6789/0 conn(0x7f3a0c13f710 legacy=0x7f3a0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.555+0000 7f3a137c6640 1 -- 172.21.15.162:0/999776155 shutdown_connections 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.555+0000 7f3a137c6640 1 -- 172.21.15.162:0/999776155 >> 172.21.15.162:0/999776155 conn(0x7f3a0c0b7ea0 msgr2=0x7f3a0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.555+0000 7f3a137c6640 1 -- 172.21.15.162:0/999776155 shutdown_connections 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.555+0000 7f3a137c6640 1 -- 172.21.15.162:0/999776155 wait complete. 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.556+0000 7f3a137c6640 10 monclient: build_initial_monmap 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.556+0000 7f3a137c6640 10 monclient: monmap: 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:54.558 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.556+0000 7f3a137c6640 1 librados: starting msgr at 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.556+0000 7f3a137c6640 1 librados: starting objecter 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.556+0000 7f3a137c6640 1 Processor -- start 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 1 -- start start 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 1 librados: setting wanted keys 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 1 librados: calling monclient init 2024-03-25T18:18:54.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 10 monclient: init 2024-03-25T18:18:54.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:54.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:54.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3a0c0a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:18:54.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3a0c13f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:18:54.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3a0c1430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:18:54.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c1ded80 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c070450 con 0x7f3a0c1430c0 2024-03-25T18:18:54.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0c06e820 con 0x7f3a0c13f710 2024-03-25T18:18:54.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:54.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.557+0000 7f3a137c6640 10 monclient(hunting): authenticate will time out at 7833.831297s 2024-03-25T18:18:54.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.561+0000 7f3a11d63640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3a0c1430c0 0x7f3a0c1e6290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:59100/0 (socket says 172.21.15.162:59100) 2024-03-25T18:18:54.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.561+0000 7f3a11d63640 1 -- 172.21.15.162:0/3076161288 learned_addr learned my addr 172.21.15.162:0/3076161288 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:18:54.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.562+0000 7f3a01ffb640 10 client.?.objecter ms_handle_connect 0x7f3a0c1430c0 2024-03-25T18:18:54.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.562+0000 7f3a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:54.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.564+0000 7f3a01ffb640 10 client.?.objecter ms_handle_connect 0x7f3a0c13f710 2024-03-25T18:18:54.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.564+0000 7f3a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:54.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.633+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533564504 0 0) 0x7f3a0c070450 con 0x7f3a0c1430c0 2024-03-25T18:18:54.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.633+0000 7f3a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:54.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.633+0000 7f3a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:54.634 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.633+0000 7f3a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:54.635 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.633+0000 7f3a01ffb640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:18:54.635 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.633+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f0004eb0 con 0x7f3a0c1430c0 2024-03-25T18:18:54.635 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.633+0000 7f3a01ffb640 10 client.?.objecter ms_handle_connect 0x7f3a0c0a98f0 2024-03-25T18:18:54.635 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.633+0000 7f3a01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:54.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.650+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851742187 0 0) 0x7f3a0c1ded80 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.650+0000 7f3a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:54.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.650+0000 7f3a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:54.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.650+0000 7f3a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:54.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.650+0000 7f3a01ffb640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:18:54.652 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.650+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a0c070450 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.654+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4242697134 0 0) 0x7f39f0004eb0 con 0x7f3a0c1430c0 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.654+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a0c1ded80 con 0x7f3a0c1430c0 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.655+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3a04001c90 con 0x7f3a0c1430c0 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.655+0000 7f3a01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.655+0000 7f3a01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.655+0000 7f3a01ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:54.656 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:54.657 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:54.657 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:54.657 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:54.657 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:54.657 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.655+0000 7f3a01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:54.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.660+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324249170 0 0) 0x7f3a0c06e820 con 0x7f3a0c13f710 2024-03-25T18:18:54.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.660+0000 7f3a01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:54.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.660+0000 7f3a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:54.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.660+0000 7f3a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:54.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.660+0000 7f3a01ffb640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:18:54.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.660+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39f0004eb0 con 0x7f3a0c13f710 2024-03-25T18:18:54.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.666+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 998770807 0 0) 0x7f3a0c070450 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.666+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a0c06e820 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.666+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3a08001870 con 0x7f3a0c0a98f0 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.666+0000 7f3a01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.666+0000 7f3a01ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.666+0000 7f3a01ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:54.668 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:54.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1718254369 0 0) 0x7f3a0c1ded80 con 0x7f3a0c1430c0 2024-03-25T18:18:54.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 >> v1:172.21.15.162:6790/0 conn(0x7f3a0c13f710 legacy=0x7f3a0c1df210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 >> v1:172.21.15.162:6789/0 conn(0x7f3a0c0a98f0 legacy=0x7f3a0c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 1 monclient: found mon.b 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c139d20 con 0x7f3a0c1430c0 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 10 monclient: _renew_subs 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c06f0a0 con 0x7f3a0c1430c0 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 10 monclient: _renew_subs 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c1dfb00 con 0x7f3a0c1430c0 2024-03-25T18:18:54.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.671+0000 7f3a137c6640 1 librados: init done 2024-03-25T18:18:54.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.672+0000 7f3a137c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:54.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.672+0000 7f3a137c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:54.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.672+0000 7f3a137c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:54.674 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.672+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c0627b0 con 0x7f3a0c1430c0 2024-03-25T18:18:54.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.965+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a04003970 con 0x7f3a0c1430c0 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.965+0000 7f3a01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.965+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3a04003d80 con 0x7f3a0c1430c0 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.965+0000 7f3a01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.965+0000 7f3a01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.965+0000 7f3a01ffb640 10 monclient: dump: 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:18:54.967 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:18:54.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.965+0000 7f3a037fe640 4 set_mon_vals no callback set 2024-03-25T18:18:54.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.966+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f3a04003fa0 con 0x7f3a0c1430c0 2024-03-25T18:18:54.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.967+0000 7f3a11562640 10 monclient: get_auth_request con 0x7f39f006dfb0 auth_method 0 2024-03-25T18:18:54.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.967+0000 7f3a01ffb640 10 client.4432.objecter ms_handle_connect 0x7f39f006dfb0 2024-03-25T18:18:54.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.971+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f3a04054a80 con 0x7f3a0c1430c0 2024-03-25T18:18:54.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.971+0000 7f3a01ffb640 10 client.4432.objecter ms_dispatch 0x7f3a0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:54.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.971+0000 7f3a01ffb640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:54.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.971+0000 7f3a01ffb640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:54.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.972+0000 7f3a01ffb640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T18:18:54.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.972+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a04050590 con 0x7f3a0c1430c0 2024-03-25T18:18:54.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.972+0000 7f3a01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:54.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:54.972+0000 7f3a01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:55.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.220+0000 7f3a137c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:55.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.220+0000 7f3a137c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:55.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.220+0000 7f3a137c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:18:55.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.220+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a0c0702c0 con 0x7f3a0c1430c0 2024-03-25T18:18:55.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.274+0000 7f3a01ffb640 1 -- 172.21.15.162:0/3076161288 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f3a040524c0 con 0x7f3a0c1430c0 2024-03-25T18:18:55.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.274+0000 7f3a01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:55.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.274+0000 7f3a01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:55.276 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:18:55.276 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:55.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 10 librados: watch_flush enter 2024-03-25T18:18:55.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 10 librados: watch_flush exit 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 10 client.4432.objecter successfully canceled tick 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 >> v1:172.21.15.191:6800/1552703795 conn(0x7f39f006dfb0 legacy=0x7f39f0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 10 monclient: shutdown 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 >> v1:172.21.15.191:6789/0 conn(0x7f3a0c1430c0 legacy=0x7f3a0c1e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 shutdown_connections 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 >> 172.21.15.162:0/3076161288 conn(0x7f3a0c0b7ea0 msgr2=0x7f3a0c13c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 shutdown_connections 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 1 -- 172.21.15.162:0/3076161288 wait complete. 2024-03-25T18:18:55.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:18:55.277+0000 7f3a137c6640 1 librados: shutdown 2024-03-25T18:18:55.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:02.096 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:02.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.214+0000 7fdda0977640 10 monclient: get_monmap_and_config 2024-03-25T18:19:02.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.214+0000 7fdda0977640 10 monclient: build_initial_monmap 2024-03-25T18:19:02.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.214+0000 7fdda0977640 10 monclient: monmap: 2024-03-25T18:19:02.216 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:02.216 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:02.216 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:19:02.215895+0000 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:19:02.215895+0000 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.215+0000 7fdda0977640 1 Processor -- start 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.215+0000 7fdda0977640 1 -- start start 2024-03-25T18:19:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.215+0000 7fdda0977640 10 monclient: init 2024-03-25T18:19:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9c0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 10 monclient(hunting): picked mon.noname-a con 0x7fdd9c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 10 monclient(hunting): picked mon.noname-b con 0x7fdd9c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:02.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c142520 con 0x7fdd9c13f710 2024-03-25T18:19:02.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c0b28a0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c0b2a30 con 0x7fdd9c1430c0 2024-03-25T18:19:02.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:02.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.216+0000 7fdda0977640 10 monclient(hunting): authenticate will time out at 7841.490512s 2024-03-25T18:19:02.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.221+0000 7fdd9ad9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fdd9c1430c0 0x7fdd9c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:56288/0 (socket says 172.21.15.162:56288) 2024-03-25T18:19:02.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.221+0000 7fdd9ad9d640 1 -- 172.21.15.162:0/1553384456 learned_addr learned my addr 172.21.15.162:0/1553384456 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:02.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.233+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37528224 0 0) 0x7fdd9c0b2a30 con 0x7fdd9c1430c0 2024-03-25T18:19:02.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.233+0000 7fdd9959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.233+0000 7fdd9959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.233+0000 7fdd9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.233+0000 7fdd9959a640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:19:02.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.233+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd84003520 con 0x7fdd9c1430c0 2024-03-25T18:19:02.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.243+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177527613 0 0) 0x7fdd9c0b28a0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.243+0000 7fdd9959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.243+0000 7fdd9959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.243+0000 7fdd9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.243+0000 7fdd9959a640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:19:02.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.243+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd9c0b2a30 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2635392088 0 0) 0x7fdd9c0b2a30 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 >> v1:172.21.15.162:6790/0 conn(0x7fdd9c1430c0 legacy=0x7fdd9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 >> v1:172.21.15.162:6789/0 conn(0x7fdd9c13f710 legacy=0x7fdd9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 1 monclient: found mon.noname-c 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c0b44c0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 10 monclient: _finish_auth 0 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdd8c001670 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdd9959a640 10 monclient: dump: 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:02.259 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdda0977640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:19:02.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.257+0000 7fdda0977640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd8c0017d0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdd9959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdd9959a640 1 -- 172.21.15.162:0/1553384456 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdd8c001be0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdda0977640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdda0977640 4 set_mon_vals no callback set 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdd9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdd9959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdd9959a640 10 monclient: dump: 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:02.276 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:02.277 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:02.277 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:02.277 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:02.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdda0977640 10 monclient: shutdown 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdda0977640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.275+0000 7fdda0977640 1 -- 172.21.15.162:0/1553384456 >> v1:172.21.15.191:6789/0 conn(0x7fdd9c0a98f0 legacy=0x7fdd9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.276+0000 7fdda0977640 1 -- 172.21.15.162:0/1553384456 shutdown_connections 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.276+0000 7fdda0977640 1 -- 172.21.15.162:0/1553384456 >> 172.21.15.162:0/1553384456 conn(0x7fdd9c0b7ea0 msgr2=0x7fdd9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.276+0000 7fdda0977640 1 -- 172.21.15.162:0/1553384456 shutdown_connections 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.276+0000 7fdda0977640 1 -- 172.21.15.162:0/1553384456 wait complete. 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 10 monclient: build_initial_monmap 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 10 monclient: monmap: 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:02.278 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:19:02.279 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:02.279 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:19:02.279 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:19:02.279 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 1 librados: starting msgr at 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 1 librados: starting objecter 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 1 Processor -- start 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 1 -- start start 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 1 librados: setting wanted keys 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 1 librados: calling monclient init 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.277+0000 7fdda0977640 10 monclient: init 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:02.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:02.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 10 monclient(hunting): picked mon.noname-b con 0x7fdd9c0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:02.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 10 monclient(hunting): picked mon.noname-a con 0x7fdd9c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:02.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 10 monclient(hunting): picked mon.noname-c con 0x7fdd9c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:02.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c137c60 con 0x7fdd9c13f710 2024-03-25T18:19:02.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c136820 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9c1369b0 con 0x7fdd9c1430c0 2024-03-25T18:19:02.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:02.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.278+0000 7fdda0977640 10 monclient(hunting): authenticate will time out at 7841.551921s 2024-03-25T18:19:02.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.286+0000 7fdd9a59c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fdd9c0a98f0 0x7fdd9c1391c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:35776/0 (socket says 172.21.15.162:35776) 2024-03-25T18:19:02.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.286+0000 7fdd9a59c640 1 -- 172.21.15.162:0/3367659196 learned_addr learned my addr 172.21.15.162:0/3367659196 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:02.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.288+0000 7fdd8affd640 10 client.?.objecter ms_handle_connect 0x7fdd9c0a98f0 2024-03-25T18:19:02.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.288+0000 7fdd8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:02.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.288+0000 7fdd8affd640 10 client.?.objecter ms_handle_connect 0x7fdd9c13f710 2024-03-25T18:19:02.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.288+0000 7fdd8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:02.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581601066 0 0) 0x7fdd9c137c60 con 0x7fdd9c13f710 2024-03-25T18:19:02.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:19:02.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd7c004eb0 con 0x7fdd9c13f710 2024-03-25T18:19:02.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376659368 0 0) 0x7fdd9c136820 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:19:02.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.298+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd9c137c60 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.300+0000 7fdd8affd640 10 client.?.objecter ms_handle_connect 0x7fdd9c1430c0 2024-03-25T18:19:02.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.300+0000 7fdd8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:02.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.315+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333980452 0 0) 0x7fdd9c1369b0 con 0x7fdd9c1430c0 2024-03-25T18:19:02.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.315+0000 7fdd8affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:02.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.315+0000 7fdd8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:02.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.315+0000 7fdd8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:02.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.315+0000 7fdd8affd640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:19:02.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.315+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd9c136820 con 0x7fdd9c1430c0 2024-03-25T18:19:02.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.319+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 859957473 0 0) 0x7fdd9c137c60 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.319+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd9c1369b0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.319+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdd8c001c20 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.319+0000 7fdd8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:02.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.319+0000 7fdd8affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:02.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.319+0000 7fdd8affd640 10 monclient(hunting): dump: 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:02.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.319+0000 7fdd8affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:02.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.324+0000 7fdd9ad9d640 1 -- 172.21.15.162:0/3367659196 >> v1:172.21.15.162:6790/0 conn(0x7fdd9c1430c0 legacy=0x7fdd9c0b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:19:02.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.324+0000 7fdd9ad9d640 1 -- 172.21.15.162:0/3367659196 >> v1:172.21.15.162:6790/0 conn(0x7fdd9c1430c0 legacy=0x7fdd9c0b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:19:02.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.324+0000 7fdd9ad9d640 1 --1- 172.21.15.162:0/3367659196 >> v1:172.21.15.162:6790/0 conn(0x7fdd9c1430c0 0x7fdd9c0b0b30 :-1 s=OPENED pgs=34288 cs=1 l=1).handle_message read tag failed 2024-03-25T18:19:02.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.324+0000 7fdd9ad9d640 1 --1- 172.21.15.162:0/3367659196 >> v1:172.21.15.162:6790/0 conn(0x7fdd9c1430c0 0x7fdd9c0b0b30 :-1 s=OPENED pgs=34288 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:19:02.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.324+0000 7fdd8affd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.162:6790/0 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.329+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894432279 0 0) 0x7fdd7c004eb0 con 0x7fdd9c13f710 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.329+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd9c137c60 con 0x7fdd9c13f710 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.329+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdd90001930 con 0x7fdd9c13f710 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.329+0000 7fdd8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.329+0000 7fdd8affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.329+0000 7fdd8affd640 10 monclient(hunting): dump: 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:02.331 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:02.332 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:02.332 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3140612399 0 0) 0x7fdd9c1369b0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 >> v1:172.21.15.162:6790/0 conn(0x7fdd9c1430c0 legacy=0x7fdd9c0b0b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 >> v1:172.21.15.162:6789/0 conn(0x7fdd9c13f710 legacy=0x7fdd9c137550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 1 monclient: found mon.b 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9c13b180 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.344 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 10 monclient: _finish_auth 0 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.342+0000 7fdd8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 10 monclient: _renew_subs 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9c13b390 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 10 monclient: _renew_subs 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9c1d7770 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.343+0000 7fdda0977640 1 librados: init done 2024-03-25T18:19:02.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.344+0000 7fdda0977640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:02.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.344+0000 7fdda0977640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:02.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.344+0000 7fdda0977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:02.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.344+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9c1398d0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.639+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd8c0038a0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.639+0000 7fdd8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.639+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdd8c003bf0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.639+0000 7fdd8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.639+0000 7fdd8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.639+0000 7fdd8affd640 10 monclient: dump: 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.640 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:02.641 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:02.641 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:02.641 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:02.641 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:02.641 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:02.641 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:02.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.639+0000 7fdd98d99640 4 set_mon_vals no callback set 2024-03-25T18:19:02.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.640+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fdd8c003e10 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.640+0000 7fdd99d9b640 10 monclient: get_auth_request con 0x7fdd7c06df10 auth_method 0 2024-03-25T18:19:02.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.641+0000 7fdd8affd640 10 client.4438.objecter ms_handle_connect 0x7fdd7c06df10 2024-03-25T18:19:02.646 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.645+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fdd8c0548a0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.646 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.645+0000 7fdd8affd640 10 client.4438.objecter ms_dispatch 0x7fdd9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:02.646 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.645+0000 7fdd8affd640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:02.646 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.645+0000 7fdd8affd640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:02.647 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.645+0000 7fdd8affd640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T18:19:02.647 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.645+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd8c0503b0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.647 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.645+0000 7fdd8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:02.647 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.645+0000 7fdd8affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:02.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.862+0000 7fdda0977640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:02.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.862+0000 7fdda0977640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:02.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.862+0000 7fdda0977640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:02.863 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.862+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd9c0627b0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.927+0000 7fdd8affd640 1 -- 172.21.15.162:0/3367659196 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fdd8c0522e0 con 0x7fdd9c0a98f0 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.927+0000 7fdd8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.927+0000 7fdd8affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:19:02.929 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:02.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 10 librados: watch_flush enter 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 10 librados: watch_flush exit 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 10 client.4438.objecter successfully canceled tick 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 >> v1:172.21.15.191:6800/1552703795 conn(0x7fdd7c06df10 legacy=0x7fdd7c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 10 monclient: shutdown 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 >> v1:172.21.15.191:6789/0 conn(0x7fdd9c0a98f0 legacy=0x7fdd9c1391c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 shutdown_connections 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 >> 172.21.15.162:0/3367659196 conn(0x7fdd9c0b7ea0 msgr2=0x7fdd9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 shutdown_connections 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 1 -- 172.21.15.162:0/3367659196 wait complete. 2024-03-25T18:19:02.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:02.930+0000 7fdda0977640 1 librados: shutdown 2024-03-25T18:19:02.942 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:07.728 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:05:19.955 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.790 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:05:20.018 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.816 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:05:20.043 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.839 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:05:20.055 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.844 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:05:20.060 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.854 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:05:20.082 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.863 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:05:20.091 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.879 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:05:20.095 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:07.885 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:05:20.112 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:09.750 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:09.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.866+0000 7f43e0f7b640 10 monclient: get_monmap_and_config 2024-03-25T18:19:09.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.866+0000 7f43e0f7b640 10 monclient: build_initial_monmap 2024-03-25T18:19:09.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.866+0000 7f43e0f7b640 10 monclient: monmap: 2024-03-25T18:19:09.868 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:09.868 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:09.868 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:19:09.867888+0000 2024-03-25T18:19:09.868 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:19:09.867888+0000 2024-03-25T18:19:09.869 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:09.869 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:09.869 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:09.869 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:19:09.869 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:19:09.869 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:09.870 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.867+0000 7f43e0f7b640 1 Processor -- start 2024-03-25T18:19:09.870 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.867+0000 7f43e0f7b640 1 -- start start 2024-03-25T18:19:09.870 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.867+0000 7f43e0f7b640 10 monclient: init 2024-03-25T18:19:09.870 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:09.870 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:09.870 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f43dc0b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:09.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f43dc0b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:09.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f43dc14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:09.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc0af940 con 0x7f43dc0b4d60 2024-03-25T18:19:09.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc0b0a70 con 0x7f43dc0b1730 2024-03-25T18:19:09.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc0b0d50 con 0x7f43dc14be90 2024-03-25T18:19:09.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:09.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.868+0000 7f43e0f7b640 10 monclient(hunting): authenticate will time out at 7849.142227s 2024-03-25T18:19:09.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.876+0000 7f43d9d9b640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f43dc0b1730 0x7f43dc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:51982/0 (socket says 172.21.15.162:51982) 2024-03-25T18:19:09.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.876+0000 7f43da59c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f43dc0b4d60 0x7f43dc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:33598/0 (socket says 172.21.15.162:33598) 2024-03-25T18:19:09.877 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.876+0000 7f43d9d9b640 1 -- 172.21.15.162:0/1130593266 learned_addr learned my addr 172.21.15.162:0/1130593266 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:09.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.883+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98452130 0 0) 0x7f43dc0b0a70 con 0x7f43dc0b1730 2024-03-25T18:19:09.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.883+0000 7f43d959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:09.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.883+0000 7f43d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:09.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.883+0000 7f43d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:09.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.883+0000 7f43d959a640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:19:09.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.883+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c4003520 con 0x7f43dc0b1730 2024-03-25T18:19:09.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.885+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956070174 0 0) 0x7f43dc0af940 con 0x7f43dc0b4d60 2024-03-25T18:19:09.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.885+0000 7f43d959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:09.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.885+0000 7f43d959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:09.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.885+0000 7f43d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:09.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.885+0000 7f43d959a640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:19:09.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.885+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43dc0b0a70 con 0x7f43dc0b4d60 2024-03-25T18:19:09.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3532038916 0 0) 0x7f43c4003520 con 0x7f43dc0b1730 2024-03-25T18:19:09.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 >> v1:172.21.15.162:6790/0 conn(0x7f43dc14be90 legacy=0x7f43dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 >> v1:172.21.15.162:6789/0 conn(0x7f43dc0b4d60 legacy=0x7f43dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:09.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 1 monclient: found mon.noname-c 2024-03-25T18:19:09.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:09.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:19:09.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc0b1030 con 0x7f43dc0b1730 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 10 monclient: _finish_auth 0 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f43d0001690 con 0x7f43dc0b1730 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43d959a640 10 monclient: dump: 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:09.895 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43e0f7b640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:19:09.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.892+0000 7f43e0f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:09.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43d00017f0 con 0x7f43dc0b1730 2024-03-25T18:19:09.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:09.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43d959a640 1 -- 172.21.15.162:0/1130593266 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f43d0001c00 con 0x7f43dc0b1730 2024-03-25T18:19:09.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:09.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43d959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:09.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43d959a640 10 monclient: dump: 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:09.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43e0f7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:09.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43e0f7b640 4 set_mon_vals no callback set 2024-03-25T18:19:09.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43e0f7b640 10 monclient: shutdown 2024-03-25T18:19:09.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43e0f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:09.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.899+0000 7f43e0f7b640 1 -- 172.21.15.162:0/1130593266 >> v1:172.21.15.191:6789/0 conn(0x7f43dc0b1730 legacy=0x7f43dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.900+0000 7f43e0f7b640 1 -- 172.21.15.162:0/1130593266 shutdown_connections 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.900+0000 7f43e0f7b640 1 -- 172.21.15.162:0/1130593266 >> 172.21.15.162:0/1130593266 conn(0x7f43dc0617a0 msgr2=0x7f43dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.900+0000 7f43e0f7b640 1 -- 172.21.15.162:0/1130593266 shutdown_connections 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.900+0000 7f43e0f7b640 1 -- 172.21.15.162:0/1130593266 wait complete. 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.900+0000 7f43e0f7b640 10 monclient: build_initial_monmap 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.900+0000 7f43e0f7b640 10 monclient: monmap: 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:19:09.905 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.901+0000 7f43e0f7b640 1 librados: starting msgr at 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.901+0000 7f43e0f7b640 1 librados: starting objecter 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.901+0000 7f43e0f7b640 1 Processor -- start 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.901+0000 7f43e0f7b640 1 -- start start 2024-03-25T18:19:09.906 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.901+0000 7f43e0f7b640 1 librados: setting wanted keys 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.901+0000 7f43e0f7b640 1 librados: calling monclient init 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.901+0000 7f43e0f7b640 10 monclient: init 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f43dc0b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f43dc0b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f43dc14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc134990 con 0x7f43dc0b4d60 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc141b50 con 0x7f43dc0b1730 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc141e30 con 0x7f43dc14be90 2024-03-25T18:19:09.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:09.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.902+0000 7f43e0f7b640 10 monclient(hunting): authenticate will time out at 7849.175906s 2024-03-25T18:19:09.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.912+0000 7f43da59c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f43dc0b1730 0x7f43dc142250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:52006/0 (socket says 172.21.15.162:52006) 2024-03-25T18:19:09.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.912+0000 7f43da59c640 1 -- 172.21.15.162:0/784518894 learned_addr learned my addr 172.21.15.162:0/784518894 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:09.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.917+0000 7f43caffd640 10 client.?.objecter ms_handle_connect 0x7f43dc0b1730 2024-03-25T18:19:09.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.917+0000 7f43caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.918+0000 7f43caffd640 10 client.?.objecter ms_handle_connect 0x7f43dc14be90 2024-03-25T18:19:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.918+0000 7f43caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:09.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.920+0000 7f43caffd640 10 client.?.objecter ms_handle_connect 0x7f43dc0b4d60 2024-03-25T18:19:09.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.920+0000 7f43caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:09.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.926+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674790817 0 0) 0x7f43dc141b50 con 0x7f43dc0b1730 2024-03-25T18:19:09.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.926+0000 7f43caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:09.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.926+0000 7f43caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:09.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.926+0000 7f43caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:09.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.926+0000 7f43caffd640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:19:09.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.926+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43bc004eb0 con 0x7f43dc0b1730 2024-03-25T18:19:09.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.929+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764891843 0 0) 0x7f43dc141e30 con 0x7f43dc14be90 2024-03-25T18:19:09.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.929+0000 7f43caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:09.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.929+0000 7f43caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:09.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.929+0000 7f43caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:09.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.929+0000 7f43caffd640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:19:09.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.929+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43dc141b50 con 0x7f43dc14be90 2024-03-25T18:19:09.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3720439567 0 0) 0x7f43bc004eb0 con 0x7f43dc0b1730 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43dc141e30 con 0x7f43dc0b1730 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f43cc0019f0 con 0x7f43dc0b1730 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 10 monclient(hunting): dump: 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:09.935 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202026225 0 0) 0x7f43dc134990 con 0x7f43dc0b4d60 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:09.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:19:09.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.933+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43bc004eb0 con 0x7f43dc0b4d60 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3326540032 0 0) 0x7f43dc141e30 con 0x7f43dc0b1730 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 >> v1:172.21.15.162:6790/0 conn(0x7f43dc14be90 legacy=0x7f43dc134280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 >> v1:172.21.15.162:6789/0 conn(0x7f43dc0b4d60 legacy=0x7f43dc133b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 1 monclient: found mon.b 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc1406d0 con 0x7f43dc0b1730 2024-03-25T18:19:09.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 10 monclient: _renew_subs 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43dc140380 con 0x7f43dc0b1730 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 10 monclient: _renew_subs 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43dc13bb10 con 0x7f43dc0b1730 2024-03-25T18:19:09.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.940+0000 7f43e0f7b640 1 librados: init done 2024-03-25T18:19:09.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.941+0000 7f43e0f7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:09.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.941+0000 7f43e0f7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:09.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.941+0000 7f43e0f7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:09.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:09.941+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43dc13bdc0 con 0x7f43dc0b1730 2024-03-25T18:19:10.159 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.158+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43cc0047e0 con 0x7f43dc0b1730 2024-03-25T18:19:10.159 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.158+0000 7f43caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:10.159 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.158+0000 7f43d8d99640 4 set_mon_vals no callback set 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.198+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f43cc004af0 con 0x7f43dc0b1730 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.198+0000 7f43caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.198+0000 7f43caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.198+0000 7f43caffd640 10 monclient: dump: 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:10.199 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:10.200 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:10.200 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:10.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.206+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f43cc01aac0 con 0x7f43dc0b1730 2024-03-25T18:19:10.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.207+0000 7f43d9d9b640 10 monclient: get_auth_request con 0x7f43bc06ddb0 auth_method 0 2024-03-25T18:19:10.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.207+0000 7f43caffd640 10 client.4444.objecter ms_handle_connect 0x7f43bc06ddb0 2024-03-25T18:19:10.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.210+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f43cc054960 con 0x7f43dc0b1730 2024-03-25T18:19:10.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.210+0000 7f43caffd640 10 client.4444.objecter ms_dispatch 0x7f43dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:10.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.210+0000 7f43caffd640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:10.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.210+0000 7f43caffd640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:10.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.210+0000 7f43caffd640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-25T18:19:10.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.213+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43cc0502a0 con 0x7f43dc0b1730 2024-03-25T18:19:10.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.213+0000 7f43caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:10.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.213+0000 7f43caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:10.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.471+0000 7f43e0f7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:10.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.472+0000 7f43e0f7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:10.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.472+0000 7f43e0f7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:10.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.472+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43dc140510 con 0x7f43dc0b1730 2024-03-25T18:19:10.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.495+0000 7f43caffd640 1 -- 172.21.15.162:0/784518894 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f43cc0523a0 con 0x7f43dc0b1730 2024-03-25T18:19:10.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.496+0000 7f43caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:10.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.496+0000 7f43caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:10.497 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:19:10.497 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.497+0000 7f43e0f7b640 10 librados: watch_flush enter 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 10 librados: watch_flush exit 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 10 client.4444.objecter successfully canceled tick 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 >> v1:172.21.15.191:6800/1552703795 conn(0x7f43bc06ddb0 legacy=0x7f43bc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 10 monclient: shutdown 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 >> v1:172.21.15.191:6789/0 conn(0x7f43dc0b1730 legacy=0x7f43dc142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 shutdown_connections 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 >> 172.21.15.162:0/784518894 conn(0x7f43dc0617a0 msgr2=0x7f43dc13c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 shutdown_connections 2024-03-25T18:19:10.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 1 -- 172.21.15.162:0/784518894 wait complete. 2024-03-25T18:19:10.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:10.498+0000 7f43e0f7b640 1 librados: shutdown 2024-03-25T18:19:10.511 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:17.318 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.432+0000 7f769dac8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.432+0000 7f769dac8640 10 monclient: build_initial_monmap 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.432+0000 7f769dac8640 10 monclient: monmap: 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:19:17.433968+0000 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:19:17.433968+0000 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:17.434 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:19:17.435 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:19:17.435 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.433+0000 7f769dac8640 1 Processor -- start 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.433+0000 7f769dac8640 1 -- start start 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.433+0000 7f769dac8640 10 monclient: init 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 10 monclient(hunting): picked mon.noname-c con 0x7f76980a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 10 monclient(hunting): picked mon.noname-a con 0x7f769813f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 10 monclient(hunting): picked mon.noname-b con 0x7f76981430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698142520 con 0x7f769813f710 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76980b28a0 con 0x7f76980a98f0 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76980b2a30 con 0x7f76981430c0 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.434+0000 7f769dac8640 10 monclient(hunting): authenticate will time out at 7856.708075s 2024-03-25T18:19:17.446 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.445+0000 7f76977fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f76980a98f0 0x7f76980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:34504/0 (socket says 172.21.15.162:34504) 2024-03-25T18:19:17.446 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.445+0000 7f76977fe640 1 -- 172.21.15.162:0/273730094 learned_addr learned my addr 172.21.15.162:0/273730094 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:17.458 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.457+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857122278 0 0) 0x7f76980b28a0 con 0x7f76980a98f0 2024-03-25T18:19:17.458 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.457+0000 7f76967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.458 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.457+0000 7f76967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.458 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.457+0000 7f76967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.457+0000 7f76967fc640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:19:17.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.457+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7680003520 con 0x7f76980a98f0 2024-03-25T18:19:17.465 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.464+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1644880370 0 0) 0x7f76980b2a30 con 0x7f76981430c0 2024-03-25T18:19:17.465 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.464+0000 7f76967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.465 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.464+0000 7f76967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.465 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.464+0000 7f76967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.466 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.464+0000 7f76967fc640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:19:17.466 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.464+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76980b28a0 con 0x7f76981430c0 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.472+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 231963716 0 0) 0x7f7680003520 con 0x7f76980a98f0 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.472+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 >> v1:172.21.15.162:6790/0 conn(0x7f76981430c0 legacy=0x7f769814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.472+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 >> v1:172.21.15.162:6789/0 conn(0x7f769813f710 legacy=0x7f76981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 1 monclient: found mon.noname-c 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76980b44c0 con 0x7f76980a98f0 2024-03-25T18:19:17.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7688001690 con 0x7f76980a98f0 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f76967fc640 10 monclient: dump: 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:17.475 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:17.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f769dac8640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:19:17.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.473+0000 7f769dac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.479+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76880017f0 con 0x7f76980a98f0 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.479+0000 7f76967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.479+0000 7f76967fc640 1 -- 172.21.15.162:0/273730094 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7688001c00 con 0x7f76980a98f0 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.479+0000 7f76967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.479+0000 7f76967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.479+0000 7f76967fc640 10 monclient: dump: 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.480 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.481 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:17.481 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:17.481 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:17.481 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:17.481 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:17.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.479+0000 7f769dac8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:17.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.479+0000 7f769dac8640 4 set_mon_vals no callback set 2024-03-25T18:19:17.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.480+0000 7f769dac8640 10 monclient: shutdown 2024-03-25T18:19:17.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.480+0000 7f769dac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:17.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.480+0000 7f769dac8640 1 -- 172.21.15.162:0/273730094 >> v1:172.21.15.191:6789/0 conn(0x7f76980a98f0 legacy=0x7f76980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.480+0000 7f769dac8640 1 -- 172.21.15.162:0/273730094 shutdown_connections 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.480+0000 7f769dac8640 1 -- 172.21.15.162:0/273730094 >> 172.21.15.162:0/273730094 conn(0x7f76980b7ea0 msgr2=0x7f76980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.480+0000 7f769dac8640 1 -- 172.21.15.162:0/273730094 shutdown_connections 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.480+0000 7f769dac8640 1 -- 172.21.15.162:0/273730094 wait complete. 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 10 monclient: build_initial_monmap 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 10 monclient: monmap: 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 1 librados: starting msgr at 2024-03-25T18:19:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 1 librados: starting objecter 2024-03-25T18:19:17.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 1 Processor -- start 2024-03-25T18:19:17.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 1 -- start start 2024-03-25T18:19:17.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 1 librados: setting wanted keys 2024-03-25T18:19:17.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 1 librados: calling monclient init 2024-03-25T18:19:17.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.481+0000 7f769dac8640 10 monclient: init 2024-03-25T18:19:17.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 10 monclient(hunting): picked mon.noname-b con 0x7f76980a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 10 monclient(hunting): picked mon.noname-c con 0x7f769813f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 10 monclient(hunting): picked mon.noname-a con 0x7f76981430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698135590 con 0x7f76981430c0 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698136e50 con 0x7f76980a98f0 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7698135070 con 0x7f769813f710 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:17.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.482+0000 7f769dac8640 10 monclient(hunting): authenticate will time out at 7856.756310s 2024-03-25T18:19:17.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.493+0000 7f76977fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f76980a98f0 0x7f7698130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:34528/0 (socket says 172.21.15.162:34528) 2024-03-25T18:19:17.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.493+0000 7f76977fe640 1 -- 172.21.15.162:0/2819897742 learned_addr learned my addr 172.21.15.162:0/2819897742 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:17.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.498+0000 7f767ffff640 10 client.?.objecter ms_handle_connect 0x7f769813f710 2024-03-25T18:19:17.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.498+0000 7f767ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.500+0000 7f767ffff640 10 client.?.objecter ms_handle_connect 0x7f76981430c0 2024-03-25T18:19:17.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.500+0000 7f767ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.502+0000 7f767ffff640 10 client.?.objecter ms_handle_connect 0x7f76980a98f0 2024-03-25T18:19:17.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.502+0000 7f767ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.504+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246801696 0 0) 0x7f7698135070 con 0x7f769813f710 2024-03-25T18:19:17.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.504+0000 7f767ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.504+0000 7f767ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.504+0000 7f767ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.504+0000 7f767ffff640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:19:17.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.504+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7684004eb0 con 0x7f769813f710 2024-03-25T18:19:17.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.517+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307669257 0 0) 0x7f7698135590 con 0x7f76981430c0 2024-03-25T18:19:17.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.518+0000 7f767ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.518+0000 7f767ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.518+0000 7f767ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.518+0000 7f767ffff640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:19:17.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.518+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7698135070 con 0x7f76981430c0 2024-03-25T18:19:17.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.520+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594761134 0 0) 0x7f7698136e50 con 0x7f76980a98f0 2024-03-25T18:19:17.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.520+0000 7f767ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.522 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.520+0000 7f767ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.522 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.520+0000 7f767ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.522 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.520+0000 7f767ffff640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:19:17.522 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.520+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7698135590 con 0x7f76980a98f0 2024-03-25T18:19:17.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.522+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956365128 0 0) 0x7f7684004eb0 con 0x7f769813f710 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.522+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7698136e50 con 0x7f769813f710 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.522+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f768c001930 con 0x7f769813f710 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.522+0000 7f767ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.522+0000 7f767ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.522+0000 7f767ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:17.524 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:17.525 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:17.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.522+0000 7f767ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.530+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 247771829 0 0) 0x7f7698135590 con 0x7f76980a98f0 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.530+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7684004eb0 con 0x7f76980a98f0 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.530+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7688001b00 con 0x7f76980a98f0 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.530+0000 7f767ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.530+0000 7f767ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.530+0000 7f767ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:17.531 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:17.532 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:17.532 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:17.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3697992613 0 0) 0x7f7684004eb0 con 0x7f76980a98f0 2024-03-25T18:19:17.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 >> v1:172.21.15.162:6790/0 conn(0x7f769813f710 legacy=0x7f76981d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 >> v1:172.21.15.162:6789/0 conn(0x7f76981430c0 legacy=0x7f76981e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:17.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 1 monclient: found mon.b 2024-03-25T18:19:17.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:17.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698135d80 con 0x7f76980a98f0 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f767ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 5 monclient: authenticate success, global_id 4450 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 10 monclient: _renew_subs 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7698131660 con 0x7f76980a98f0 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 10 monclient: _renew_subs 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76980b78a0 con 0x7f76980a98f0 2024-03-25T18:19:17.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.539+0000 7f769dac8640 1 librados: init done 2024-03-25T18:19:17.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.540+0000 7f769dac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.540+0000 7f769dac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.540+0000 7f769dac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:17.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.540+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7698135f70 con 0x7f76980a98f0 2024-03-25T18:19:17.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.807+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7688001220 con 0x7f76980a98f0 2024-03-25T18:19:17.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.807+0000 7f767ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.807+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7688004a90 con 0x7f76980a98f0 2024-03-25T18:19:17.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.807+0000 7f767ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.807+0000 7f767ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:17.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.807+0000 7f767ffff640 10 monclient: dump: 2024-03-25T18:19:17.808 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:17.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.807+0000 7f7695ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:17.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.808+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f768801aa60 con 0x7f76980a98f0 2024-03-25T18:19:17.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.808+0000 7f7696ffd640 10 monclient: get_auth_request con 0x7f768406df60 auth_method 0 2024-03-25T18:19:17.810 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.809+0000 7f767ffff640 10 client.4450.objecter ms_handle_connect 0x7f768406df60 2024-03-25T18:19:17.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.813+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f76880546d0 con 0x7f76980a98f0 2024-03-25T18:19:17.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.813+0000 7f767ffff640 10 client.4450.objecter ms_dispatch 0x7f7698000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:17.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.813+0000 7f767ffff640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:17.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.813+0000 7f767ffff640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:17.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.813+0000 7f767ffff640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T18:19:17.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.813+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7688050010 con 0x7f76980a98f0 2024-03-25T18:19:17.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.813+0000 7f767ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:17.813+0000 7f767ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:18.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.070+0000 7f769dac8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:18.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.070+0000 7f769dac8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:18.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.070+0000 7f769dac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:18.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.070+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7698135200 con 0x7f76980a98f0 2024-03-25T18:19:18.097 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.096+0000 7f767ffff640 1 -- 172.21.15.162:0/2819897742 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f7688052110 con 0x7f76980a98f0 2024-03-25T18:19:18.097 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.096+0000 7f767ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:18.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.096+0000 7f767ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:18.098 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:19:18.098 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:18.100 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.098+0000 7f769dac8640 10 librados: watch_flush enter 2024-03-25T18:19:18.100 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.098+0000 7f769dac8640 10 librados: watch_flush exit 2024-03-25T18:19:18.100 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.098+0000 7f769dac8640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T18:19:18.100 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.098+0000 7f769dac8640 10 client.4450.objecter successfully canceled tick 2024-03-25T18:19:18.100 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 >> v1:172.21.15.191:6800/1552703795 conn(0x7f768406df60 legacy=0x7f7684070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 10 monclient: shutdown 2024-03-25T18:19:18.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:18.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 >> v1:172.21.15.191:6789/0 conn(0x7f76980a98f0 legacy=0x7f7698130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 shutdown_connections 2024-03-25T18:19:18.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 >> 172.21.15.162:0/2819897742 conn(0x7f76980b7ea0 msgr2=0x7f7698074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:18.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 shutdown_connections 2024-03-25T18:19:18.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 1 -- 172.21.15.162:0/2819897742 wait complete. 2024-03-25T18:19:18.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:18.099+0000 7f769dac8640 1 librados: shutdown 2024-03-25T18:19:18.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:24.919 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.040+0000 7f809884a640 10 monclient: get_monmap_and_config 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.040+0000 7f809884a640 10 monclient: build_initial_monmap 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.040+0000 7f809884a640 10 monclient: monmap: 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:19:25.041265+0000 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:19:25.041265+0000 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:25.041 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:25.042 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:19:25.042 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:19:25.042 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 1 Processor -- start 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 1 -- start start 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 10 monclient: init 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 10 monclient(hunting): picked mon.noname-a con 0x7f80940a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 10 monclient(hunting): picked mon.noname-b con 0x7f809413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 10 monclient(hunting): picked mon.noname-c con 0x7f80941430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8094142520 con 0x7f80940a98f0 2024-03-25T18:19:25.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940b28a0 con 0x7f80941430c0 2024-03-25T18:19:25.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940b2a30 con 0x7f809413f710 2024-03-25T18:19:25.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:25.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.041+0000 7f809884a640 10 monclient(hunting): authenticate will time out at 7864.315465s 2024-03-25T18:19:25.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.051+0000 7f8092d9d640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f80941430c0 0x7f809414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:35122/0 (socket says 172.21.15.162:35122) 2024-03-25T18:19:25.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.051+0000 7f8092d9d640 1 -- 172.21.15.162:0/535924481 learned_addr learned my addr 172.21.15.162:0/535924481 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:25.061 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.060+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934346621 0 0) 0x7f80940b28a0 con 0x7f80941430c0 2024-03-25T18:19:25.061 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.060+0000 7f8091d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.061 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.060+0000 7f8091d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.061 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.060+0000 7f8091d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.061 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.060+0000 7f8091d9b640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:19:25.062 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.060+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8080003520 con 0x7f80941430c0 2024-03-25T18:19:25.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.066+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958882344 0 0) 0x7f80940b2a30 con 0x7f809413f710 2024-03-25T18:19:25.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.066+0000 7f8091d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.066+0000 7f8091d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.066+0000 7f8091d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.066+0000 7f8091d9b640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:19:25.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.066+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80940b28a0 con 0x7f809413f710 2024-03-25T18:19:25.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.070+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614634515 0 0) 0x7f8094142520 con 0x7f80940a98f0 2024-03-25T18:19:25.071 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.070+0000 7f8091d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.072 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.070+0000 7f8091d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.072 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.070+0000 7f8091d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.072 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.070+0000 7f8091d9b640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:19:25.072 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.070+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80940b2a30 con 0x7f80940a98f0 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2512163515 0 0) 0x7f8080003520 con 0x7f80941430c0 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 >> v1:172.21.15.162:6790/0 conn(0x7f809413f710 legacy=0x7f80941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 >> v1:172.21.15.162:6789/0 conn(0x7f80940a98f0 legacy=0x7f80940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 1 monclient: found mon.noname-c 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940b44c0 con 0x7f80941430c0 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 10 monclient: _finish_auth 0 2024-03-25T18:19:25.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f808c001a10 con 0x7f80941430c0 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f8091d9b640 10 monclient: dump: 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:25.081 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:25.082 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:25.082 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:25.082 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:25.082 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f809884a640 5 monclient: authenticate success, global_id 4453 2024-03-25T18:19:25.082 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.078+0000 7f809884a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:25.083 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.081+0000 7f8091d9b640 1 -- 172.21.15.162:0/535924481 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f808c001b70 con 0x7f80941430c0 2024-03-25T18:19:25.083 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.081+0000 7f8091d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:25.083 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.081+0000 7f809884a640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:25.083 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.081+0000 7f809884a640 4 set_mon_vals no callback set 2024-03-25T18:19:25.083 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.081+0000 7f809884a640 10 monclient: shutdown 2024-03-25T18:19:25.083 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.081+0000 7f809884a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:25.083 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.081+0000 7f809884a640 1 -- 172.21.15.162:0/535924481 >> v1:172.21.15.191:6789/0 conn(0x7f80941430c0 legacy=0x7f809414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.082+0000 7f809884a640 1 -- 172.21.15.162:0/535924481 shutdown_connections 2024-03-25T18:19:25.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.082+0000 7f809884a640 1 -- 172.21.15.162:0/535924481 >> 172.21.15.162:0/535924481 conn(0x7f80940b7ea0 msgr2=0x7f80940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:25.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.082+0000 7f809884a640 1 -- 172.21.15.162:0/535924481 shutdown_connections 2024-03-25T18:19:25.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.082+0000 7f809884a640 1 -- 172.21.15.162:0/535924481 wait complete. 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.082+0000 7f809884a640 10 monclient: build_initial_monmap 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.082+0000 7f809884a640 10 monclient: monmap: 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:19:25.086 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.083+0000 7f809884a640 1 librados: starting msgr at 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.083+0000 7f809884a640 1 librados: starting objecter 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.083+0000 7f809884a640 1 Processor -- start 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.083+0000 7f809884a640 1 -- start start 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.083+0000 7f809884a640 1 librados: setting wanted keys 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.083+0000 7f809884a640 1 librados: calling monclient init 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.083+0000 7f809884a640 10 monclient: init 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 10 monclient(hunting): picked mon.noname-b con 0x7f80940a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 10 monclient(hunting): picked mon.noname-c con 0x7f809413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:25.087 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 10 monclient(hunting): picked mon.noname-a con 0x7f80941430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:25.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8094138ca0 con 0x7f80941430c0 2024-03-25T18:19:25.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8094134620 con 0x7f80940a98f0 2024-03-25T18:19:25.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80941347b0 con 0x7f809413f710 2024-03-25T18:19:25.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:25.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.084+0000 7f809884a640 10 monclient(hunting): authenticate will time out at 7864.358089s 2024-03-25T18:19:25.094 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.093+0000 7f808bfff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f809413f710 0x7f8094137b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:60030/0 (socket says 172.21.15.162:60030) 2024-03-25T18:19:25.094 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.093+0000 7f808bfff640 1 -- 172.21.15.162:0/3493956581 learned_addr learned my addr 172.21.15.162:0/3493956581 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:25.096 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.095+0000 7f808affd640 10 client.?.objecter ms_handle_connect 0x7f809413f710 2024-03-25T18:19:25.096 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.095+0000 7f808affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:25.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.097+0000 7f808affd640 10 client.?.objecter ms_handle_connect 0x7f80940a98f0 2024-03-25T18:19:25.098 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.097+0000 7f808affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:25.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.105+0000 7f808affd640 10 client.?.objecter ms_handle_connect 0x7f80941430c0 2024-03-25T18:19:25.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.105+0000 7f808affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:25.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.106+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607739144 0 0) 0x7f80941347b0 con 0x7f809413f710 2024-03-25T18:19:25.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.106+0000 7f808affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.106+0000 7f808affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.106+0000 7f808affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.106+0000 7f808affd640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:19:25.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.106+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8074004eb0 con 0x7f809413f710 2024-03-25T18:19:25.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.108+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44585085 0 0) 0x7f8094134620 con 0x7f80940a98f0 2024-03-25T18:19:25.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.108+0000 7f808affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.108+0000 7f808affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.108+0000 7f808affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.108+0000 7f808affd640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:19:25.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.108+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80941347b0 con 0x7f80940a98f0 2024-03-25T18:19:25.116 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.115+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722117858 0 0) 0x7f8094138ca0 con 0x7f80941430c0 2024-03-25T18:19:25.116 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.115+0000 7f808affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:25.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.115+0000 7f808affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:25.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.115+0000 7f808affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:25.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.115+0000 7f808affd640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:19:25.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.115+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8094134620 con 0x7f80941430c0 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.121+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3820958764 0 0) 0x7f8074004eb0 con 0x7f809413f710 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.121+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8094138ca0 con 0x7f809413f710 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.121+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f807c001a30 con 0x7f809413f710 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.121+0000 7f808affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.122+0000 7f808affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.122+0000 7f808affd640 10 monclient(hunting): dump: 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:25.123 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:25.124 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:25.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.122+0000 7f808affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:25.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.134+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4065828055 0 0) 0x7f8094134620 con 0x7f80941430c0 2024-03-25T18:19:25.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.134+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8074004eb0 con 0x7f80941430c0 2024-03-25T18:19:25.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.134+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f808c001e80 con 0x7f80941430c0 2024-03-25T18:19:25.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.134+0000 7f808affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:25.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.134+0000 7f808affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:25.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.134+0000 7f808affd640 10 monclient(hunting): dump: 2024-03-25T18:19:25.135 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:25.136 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:25.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.139+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2473359401 0 0) 0x7f80941347b0 con 0x7f80940a98f0 2024-03-25T18:19:25.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.139+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8094134620 con 0x7f80940a98f0 2024-03-25T18:19:25.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.139+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8084001870 con 0x7f80940a98f0 2024-03-25T18:19:25.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.139+0000 7f808affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:25.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.139+0000 7f808affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:25.140 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.139+0000 7f808affd640 10 monclient(hunting): dump: 2024-03-25T18:19:25.140 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:25.141 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:25.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 878320920 0 0) 0x7f8094138ca0 con 0x7f809413f710 2024-03-25T18:19:25.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 >> v1:172.21.15.191:6789/0 conn(0x7f80940a98f0 legacy=0x7f80941383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 >> v1:172.21.15.162:6789/0 conn(0x7f80941430c0 legacy=0x7f8094133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:25.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 1 monclient: found mon.c 2024-03-25T18:19:25.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:25.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:25.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8094134940 con 0x7f809413f710 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 10 monclient: _finish_auth 0 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f808affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 5 monclient: authenticate success, global_id 4460 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 10 monclient: _renew_subs 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80941d7590 con 0x7f809413f710 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 10 monclient: _renew_subs 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80941d7a70 con 0x7f809413f710 2024-03-25T18:19:25.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.145+0000 7f809884a640 1 librados: init done 2024-03-25T18:19:25.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.147+0000 7f809884a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:25.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.147+0000 7f809884a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:25.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.147+0000 7f809884a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:25.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.147+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8094138f80 con 0x7f809413f710 2024-03-25T18:19:25.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.474+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f807c001c70 con 0x7f809413f710 2024-03-25T18:19:25.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.474+0000 7f808affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:25.475 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.474+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f807c003a60 con 0x7f809413f710 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.474+0000 7f808affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.474+0000 7f808affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.474+0000 7f808affd640 10 monclient: dump: 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:25.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.474+0000 7f809159a640 4 set_mon_vals no callback set 2024-03-25T18:19:25.477 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.475+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f807c003c40 con 0x7f809413f710 2024-03-25T18:19:25.477 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.475+0000 7f809259c640 10 monclient: get_auth_request con 0x7f807406e0f0 auth_method 0 2024-03-25T18:19:25.477 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.476+0000 7f808affd640 10 client.4460.objecter ms_handle_connect 0x7f807406e0f0 2024-03-25T18:19:25.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.480+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f807c054840 con 0x7f809413f710 2024-03-25T18:19:25.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.480+0000 7f808affd640 10 client.4460.objecter ms_dispatch 0x7f8094000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:25.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.480+0000 7f808affd640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:25.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.480+0000 7f808affd640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:25.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.480+0000 7f808affd640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-25T18:19:25.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.480+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f807c050350 con 0x7f809413f710 2024-03-25T18:19:25.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.480+0000 7f808affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:25.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.480+0000 7f808affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:25.705 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.704+0000 7f809884a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:25.705 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.704+0000 7f809884a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:25.705 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.704+0000 7f809884a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:25.705 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.704+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8094074cc0 con 0x7f809413f710 2024-03-25T18:19:25.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.755+0000 7f808affd640 1 -- 172.21.15.162:0/3493956581 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f807c052280 con 0x7f809413f710 2024-03-25T18:19:25.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.755+0000 7f808affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:25.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.755+0000 7f808affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:25.756 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:19:25.757 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:25.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 10 librados: watch_flush enter 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 10 librados: watch_flush exit 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 10 client.4460.objecter successfully canceled tick 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 >> v1:172.21.15.191:6800/1552703795 conn(0x7f807406e0f0 legacy=0x7f80740705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 10 monclient: shutdown 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 >> v1:172.21.15.162:6790/0 conn(0x7f809413f710 legacy=0x7f8094137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 shutdown_connections 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 >> 172.21.15.162:0/3493956581 conn(0x7f80940b7ea0 msgr2=0x7f8094074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 shutdown_connections 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 1 -- 172.21.15.162:0/3493956581 wait complete. 2024-03-25T18:19:25.760 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:25.758+0000 7f809884a640 1 librados: shutdown 2024-03-25T18:19:25.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:27.861 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:05:40.088 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.874 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:05:40.101 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.907 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:05:40.134 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.912 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:05:40.140 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.927 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:05:40.155 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.936 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:05:40.152 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.940 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:05:40.156 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.947 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:05:40.175 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:27.960 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:05:40.176 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:32.578 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:32.681 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.681+0000 7fbdbddf4640 10 monclient: get_monmap_and_config 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.681+0000 7fbdbddf4640 10 monclient: build_initial_monmap 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.681+0000 7fbdbddf4640 10 monclient: monmap: 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:19:32.682104+0000 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:19:32.682104+0000 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:19:32.682 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.681+0000 7fbdbddf4640 1 Processor -- start 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.681+0000 7fbdbddf4640 1 -- start start 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.681+0000 7fbdbddf4640 10 monclient: init 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 10 monclient(hunting): picked mon.noname-b con 0x7fbdb80b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 10 monclient(hunting): picked mon.noname-a con 0x7fbdb80b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 10 monclient(hunting): picked mon.noname-c con 0x7fbdb814be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb80af940 con 0x7fbdb80b1730 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb80b0a70 con 0x7fbdb814be90 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb80b0d50 con 0x7fbdb80b4d60 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdbddf4640 10 monclient(hunting): authenticate will time out at 7871.956071s 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.682+0000 7fbdb77fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fbdb80b4d60 0x7fbdb813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:36462/0 (socket says 172.21.15.162:36462) 2024-03-25T18:19:32.684 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.683+0000 7fbdb77fe640 1 -- 172.21.15.162:0/350085701 learned_addr learned my addr 172.21.15.162:0/350085701 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:32.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.699+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305429581 0 0) 0x7fbdb80b0a70 con 0x7fbdb814be90 2024-03-25T18:19:32.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.699+0000 7fbdb67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:32.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.699+0000 7fbdb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:32.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.699+0000 7fbdb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:32.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.699+0000 7fbdb67fc640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:19:32.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.699+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda0003520 con 0x7fbdb814be90 2024-03-25T18:19:32.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408902735 0 0) 0x7fbdb80af940 con 0x7fbdb80b1730 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb80b0a70 con 0x7fbdb80b1730 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3720971511 0 0) 0x7fbda0003520 con 0x7fbdb814be90 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 >> v1:172.21.15.162:6790/0 conn(0x7fbdb80b4d60 legacy=0x7fbdb813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 >> v1:172.21.15.162:6789/0 conn(0x7fbdb80b1730 legacy=0x7fbdb80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 1 monclient: found mon.noname-c 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb80b1030 con 0x7fbdb814be90 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fbdb0001a10 con 0x7fbdb814be90 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdbddf4640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdbddf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdb67fc640 10 monclient: dump: 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.709+0000 7fbdbddf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:32.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.726+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdb0001b70 con 0x7fbdb814be90 2024-03-25T18:19:32.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.726+0000 7fbdb67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:32.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.726+0000 7fbdb67fc640 1 -- 172.21.15.162:0/350085701 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fbdb0001f80 con 0x7fbdb814be90 2024-03-25T18:19:32.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.726+0000 7fbdbddf4640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:32.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.726+0000 7fbdbddf4640 4 set_mon_vals no callback set 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.726+0000 7fbdb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.726+0000 7fbdb67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.726+0000 7fbdb67fc640 10 monclient: dump: 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:32.728 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 10 monclient: shutdown 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 1 -- 172.21.15.162:0/350085701 >> v1:172.21.15.191:6789/0 conn(0x7fbdb814be90 legacy=0x7fbdb814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 1 -- 172.21.15.162:0/350085701 shutdown_connections 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 1 -- 172.21.15.162:0/350085701 >> 172.21.15.162:0/350085701 conn(0x7fbdb80617a0 msgr2=0x7fbdb8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 1 -- 172.21.15.162:0/350085701 shutdown_connections 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 1 -- 172.21.15.162:0/350085701 wait complete. 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 10 monclient: build_initial_monmap 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.727+0000 7fbdbddf4640 10 monclient: monmap: 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:32.730 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.728+0000 7fbdbddf4640 1 librados: starting msgr at 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.728+0000 7fbdbddf4640 1 librados: starting objecter 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.728+0000 7fbdbddf4640 1 Processor -- start 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.728+0000 7fbdbddf4640 1 -- start start 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.728+0000 7fbdbddf4640 1 librados: setting wanted keys 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.728+0000 7fbdbddf4640 1 librados: calling monclient init 2024-03-25T18:19:32.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.728+0000 7fbdbddf4640 10 monclient: init 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 10 monclient(hunting): picked mon.noname-b con 0x7fbdb80b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 10 monclient(hunting): picked mon.noname-c con 0x7fbdb80b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 10 monclient(hunting): picked mon.noname-a con 0x7fbdb814be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb8138120 con 0x7fbdb814be90 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb8137c40 con 0x7fbdb80b1730 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb8137dd0 con 0x7fbdb80b4d60 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:32.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.729+0000 7fbdbddf4640 10 monclient(hunting): authenticate will time out at 7872.002890s 2024-03-25T18:19:32.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.739+0000 7fbdb77fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fbdb80b1730 0x7fbdb8140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:39718/0 (socket says 172.21.15.162:39718) 2024-03-25T18:19:32.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.740+0000 7fbdb77fe640 1 -- 172.21.15.162:0/2835819520 learned_addr learned my addr 172.21.15.162:0/2835819520 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:32.742 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.742+0000 7fbd9ffff640 10 client.?.objecter ms_handle_connect 0x7fbdb80b1730 2024-03-25T18:19:32.742 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.742+0000 7fbd9ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:32.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.745+0000 7fbd9ffff640 10 client.?.objecter ms_handle_connect 0x7fbdb814be90 2024-03-25T18:19:32.746 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.745+0000 7fbd9ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:32.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.747+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899627671 0 0) 0x7fbdb8137c40 con 0x7fbdb80b1730 2024-03-25T18:19:32.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.747+0000 7fbd9ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:32.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.747+0000 7fbd9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:32.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.747+0000 7fbd9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:32.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.747+0000 7fbd9ffff640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:19:32.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.747+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbda4004eb0 con 0x7fbdb80b1730 2024-03-25T18:19:32.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.755+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808622398 0 0) 0x7fbdb8138120 con 0x7fbdb814be90 2024-03-25T18:19:32.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.755+0000 7fbd9ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:32.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.755+0000 7fbd9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:32.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.755+0000 7fbd9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:32.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.755+0000 7fbd9ffff640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:19:32.757 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.755+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb8137c40 con 0x7fbdb814be90 2024-03-25T18:19:32.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.761+0000 7fbd9ffff640 10 client.?.objecter ms_handle_connect 0x7fbdb80b4d60 2024-03-25T18:19:32.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.761+0000 7fbd9ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.765+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 907415227 0 0) 0x7fbda4004eb0 con 0x7fbdb80b1730 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.765+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdb8138120 con 0x7fbdb80b1730 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.765+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fbda80018b0 con 0x7fbdb80b1730 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.765+0000 7fbd9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.765+0000 7fbd9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.765+0000 7fbd9ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:32.767 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:32.768 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:32.768 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:32.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.765+0000 7fbd9ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:32.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2926524461 0 0) 0x7fbdb8138120 con 0x7fbdb80b1730 2024-03-25T18:19:32.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 >> v1:172.21.15.162:6790/0 conn(0x7fbdb80b4d60 legacy=0x7fbdb8131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 >> v1:172.21.15.162:6789/0 conn(0x7fbdb814be90 legacy=0x7fbdb8142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:32.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 1 monclient: found mon.b 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8132620 con 0x7fbdb80b1730 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbd9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 10 monclient: _renew_subs 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb8144f00 con 0x7fbdb80b1730 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 10 monclient: _renew_subs 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb8142e20 con 0x7fbdb80b1730 2024-03-25T18:19:32.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.775+0000 7fbdbddf4640 1 librados: init done 2024-03-25T18:19:32.778 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.776+0000 7fbdbddf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:32.778 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.776+0000 7fbdbddf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:32.778 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.776+0000 7fbdbddf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:32.778 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:32.776+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb81452a0 con 0x7fbdb80b1730 2024-03-25T18:19:33.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.050+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbda80047e0 con 0x7fbdb80b1730 2024-03-25T18:19:33.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.050+0000 7fbd9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:33.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.050+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fbda8004c30 con 0x7fbdb80b1730 2024-03-25T18:19:33.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.050+0000 7fbd9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:33.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.050+0000 7fbd9ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:33.051 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.050+0000 7fbd9ffff640 10 monclient: dump: 2024-03-25T18:19:33.051 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:33.052 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.050+0000 7fbdb5ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:33.053 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.051+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fbda8004e50 con 0x7fbdb80b1730 2024-03-25T18:19:33.053 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.051+0000 7fbdb6ffd640 10 monclient: get_auth_request con 0x7fbda406dd10 auth_method 0 2024-03-25T18:19:33.053 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.051+0000 7fbd9ffff640 10 client.4462.objecter ms_handle_connect 0x7fbda406dd10 2024-03-25T18:19:33.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.057+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fbda8054870 con 0x7fbdb80b1730 2024-03-25T18:19:33.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.057+0000 7fbd9ffff640 10 client.4462.objecter ms_dispatch 0x7fbdb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:33.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.057+0000 7fbd9ffff640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:33.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.057+0000 7fbd9ffff640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:33.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.057+0000 7fbd9ffff640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T18:19:33.059 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.058+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbda8050380 con 0x7fbdb80b1730 2024-03-25T18:19:33.059 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.058+0000 7fbd9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:33.059 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.058+0000 7fbd9ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:33.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.310+0000 7fbdbddf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:33.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.310+0000 7fbdbddf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:33.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.310+0000 7fbdbddf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:33.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.310+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdb8141ee0 con 0x7fbdb80b1730 2024-03-25T18:19:33.339 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.337+0000 7fbd9ffff640 1 -- 172.21.15.162:0/2835819520 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fbda80522b0 con 0x7fbdb80b1730 2024-03-25T18:19:33.339 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.337+0000 7fbd9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:33.339 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.337+0000 7fbd9ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:33.339 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:19:33.339 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:33.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 10 librados: watch_flush enter 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 10 librados: watch_flush exit 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 10 client.4462.objecter successfully canceled tick 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 >> v1:172.21.15.191:6800/1552703795 conn(0x7fbda406dd10 legacy=0x7fbda40701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 10 monclient: shutdown 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 >> v1:172.21.15.191:6789/0 conn(0x7fbdb80b1730 legacy=0x7fbdb8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 shutdown_connections 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 >> 172.21.15.162:0/2835819520 conn(0x7fbdb80617a0 msgr2=0x7fbdb80b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 shutdown_connections 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 1 -- 172.21.15.162:0/2835819520 wait complete. 2024-03-25T18:19:33.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:33.340+0000 7fbdbddf4640 1 librados: shutdown 2024-03-25T18:19:33.353 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:40.159 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.267+0000 7f531f64a640 10 monclient: get_monmap_and_config 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.267+0000 7f531f64a640 10 monclient: build_initial_monmap 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.268+0000 7f531f64a640 10 monclient: monmap: 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:19:40.269023+0000 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:19:40.269023+0000 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:19:40.269 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:19:40.270 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:40.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.268+0000 7f531f64a640 1 Processor -- start 2024-03-25T18:19:40.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.269+0000 7f531f64a640 1 -- start start 2024-03-25T18:19:40.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.269+0000 7f531f64a640 10 monclient: init 2024-03-25T18:19:40.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.269+0000 7f531f64a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:40.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.269+0000 7f531f64a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:40.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.269+0000 7f531f64a640 10 monclient(hunting): picked mon.noname-c con 0x7f531806c380 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:40.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.269+0000 7f531f64a640 10 monclient(hunting): picked mon.noname-a con 0x7f531806d670 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:40.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.269+0000 7f531f64a640 10 monclient(hunting): picked mon.noname-b con 0x7f53180af8f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:40.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.269+0000 7f531f64a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f531806c9a0 con 0x7f531806d670 2024-03-25T18:19:40.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.270+0000 7f531f64a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180ada90 con 0x7f531806c380 2024-03-25T18:19:40.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.270+0000 7f531f64a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53180b0d10 con 0x7f53180af8f0 2024-03-25T18:19:40.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.270+0000 7f531f64a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:40.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.270+0000 7f531f64a640 10 monclient(hunting): authenticate will time out at 7879.543806s 2024-03-25T18:19:40.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.275+0000 7f531d3e6640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f531806c380 0x7f53180b0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:42888/0 (socket says 172.21.15.162:42888) 2024-03-25T18:19:40.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.275+0000 7f531d3e6640 1 -- 172.21.15.162:0/3819764933 learned_addr learned my addr 172.21.15.162:0/3819764933 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:40.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.288+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049448553 0 0) 0x7f53180ada90 con 0x7f531806c380 2024-03-25T18:19:40.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.288+0000 7f530ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.288+0000 7f530ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.288+0000 7f530ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.288+0000 7f530ffff640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:19:40.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.288+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5304003520 con 0x7f531806c380 2024-03-25T18:19:40.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.294+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039053685 0 0) 0x7f53180b0d10 con 0x7f53180af8f0 2024-03-25T18:19:40.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.294+0000 7f530ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.294+0000 7f530ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.294+0000 7f530ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.294+0000 7f530ffff640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:19:40.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.294+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53180ada90 con 0x7f53180af8f0 2024-03-25T18:19:40.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 892931690 0 0) 0x7f5304003520 con 0x7f531806c380 2024-03-25T18:19:40.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 >> v1:172.21.15.162:6790/0 conn(0x7f53180af8f0 legacy=0x7f53180aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 >> v1:172.21.15.162:6789/0 conn(0x7f531806d670 legacy=0x7f53180ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:40.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 1 monclient: found mon.noname-c 2024-03-25T18:19:40.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:40.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53180add00 con 0x7f531806c380 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5314001690 con 0x7f531806c380 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f531f64a640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f531f64a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f530ffff640 10 monclient: dump: 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:40.301 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.302 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.302 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.302 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:40.302 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:40.302 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:40.302 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:40.302 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:40.302 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.298+0000 7f531f64a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.307+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53140017f0 con 0x7f531806c380 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.307+0000 7f530ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.307+0000 7f530ffff640 1 -- 172.21.15.162:0/3819764933 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5314001b80 con 0x7f531806c380 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.307+0000 7f530ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.307+0000 7f530ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.307+0000 7f530ffff640 10 monclient: dump: 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.308 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.309 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.309 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:40.309 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:40.309 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:40.309 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:40.309 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:40.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.307+0000 7f531f64a640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:40.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.307+0000 7f531f64a640 4 set_mon_vals no callback set 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.308+0000 7f531f64a640 10 monclient: shutdown 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.308+0000 7f531f64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.308+0000 7f531f64a640 1 -- 172.21.15.162:0/3819764933 >> v1:172.21.15.191:6789/0 conn(0x7f531806c380 legacy=0x7f53180b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.308+0000 7f531f64a640 1 -- 172.21.15.162:0/3819764933 shutdown_connections 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.308+0000 7f531f64a640 1 -- 172.21.15.162:0/3819764933 >> 172.21.15.162:0/3819764933 conn(0x7f53180b5c00 msgr2=0x7f53180b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.308+0000 7f531f64a640 1 -- 172.21.15.162:0/3819764933 shutdown_connections 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.308+0000 7f531f64a640 1 -- 172.21.15.162:0/3819764933 wait complete. 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 10 monclient: build_initial_monmap 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 10 monclient: monmap: 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:40.311 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 1 librados: starting msgr at 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 1 librados: starting objecter 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 1 Processor -- start 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 1 -- start start 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 1 librados: setting wanted keys 2024-03-25T18:19:40.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 1 librados: calling monclient init 2024-03-25T18:19:40.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.309+0000 7f531f64a640 10 monclient: init 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 10 monclient(hunting): picked mon.noname-a con 0x7f531806c380 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 10 monclient(hunting): picked mon.noname-b con 0x7f531806d670 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 10 monclient(hunting): picked mon.noname-c con 0x7f53180af8f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5318144c50 con 0x7f531806c380 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5318144f30 con 0x7f531806d670 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53181453c0 con 0x7f53180af8f0 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:40.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.310+0000 7f531f64a640 10 monclient(hunting): authenticate will time out at 7879.584570s 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.321+0000 7f531dbe7640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f53180af8f0 0x7f53181e2530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:60290/0 (socket says 172.21.15.162:60290) 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.321+0000 7f531dbe7640 1 -- 172.21.15.162:0/3429737468 learned_addr learned my addr 172.21.15.162:0/3429737468 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:40.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.324+0000 7f530dffb640 10 client.?.objecter ms_handle_connect 0x7f53180af8f0 2024-03-25T18:19:40.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.324+0000 7f530dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:40.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.330+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257050561 0 0) 0x7f53181453c0 con 0x7f53180af8f0 2024-03-25T18:19:40.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.330+0000 7f530dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.330+0000 7f530dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.330+0000 7f530dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.330+0000 7f530dffb640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:19:40.331 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.330+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52fc004eb0 con 0x7f53180af8f0 2024-03-25T18:19:40.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.341+0000 7f530dffb640 10 client.?.objecter ms_handle_connect 0x7f531806d670 2024-03-25T18:19:40.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.341+0000 7f530dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:40.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.342+0000 7f530dffb640 10 client.?.objecter ms_handle_connect 0x7f531806c380 2024-03-25T18:19:40.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.342+0000 7f530dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:40.351 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.350+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956538760 0 0) 0x7f5318144f30 con 0x7f531806d670 2024-03-25T18:19:40.351 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.350+0000 7f530dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.352 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.350+0000 7f530dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.352 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.350+0000 7f530dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.352 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.350+0000 7f530dffb640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:19:40.352 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.350+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53181453c0 con 0x7f531806d670 2024-03-25T18:19:40.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.353+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2996049867 0 0) 0x7f52fc004eb0 con 0x7f53180af8f0 2024-03-25T18:19:40.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.353+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5318144f30 con 0x7f53180af8f0 2024-03-25T18:19:40.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.353+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5310001c90 con 0x7f53180af8f0 2024-03-25T18:19:40.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.353+0000 7f530dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.353+0000 7f530dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.353+0000 7f530dffb640 10 monclient(hunting): dump: 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:40.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.353+0000 7f530dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:40.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.362+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3464812396 0 0) 0x7f53181453c0 con 0x7f531806d670 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.362+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52fc004eb0 con 0x7f531806d670 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.362+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f53080018b0 con 0x7f531806d670 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.362+0000 7f530dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.362+0000 7f530dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.362+0000 7f530dffb640 10 monclient(hunting): dump: 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:40.364 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:40.365 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:40.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.370+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810219487 0 0) 0x7f5318144c50 con 0x7f531806c380 2024-03-25T18:19:40.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.370+0000 7f530dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:40.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.370+0000 7f530dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:40.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.370+0000 7f530dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:40.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.370+0000 7f530dffb640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:19:40.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.370+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53181453c0 con 0x7f531806c380 2024-03-25T18:19:40.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.378+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 212790462 0 0) 0x7f52fc004eb0 con 0x7f531806d670 2024-03-25T18:19:40.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 >> v1:172.21.15.162:6790/0 conn(0x7f53180af8f0 legacy=0x7f53181e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 >> v1:172.21.15.162:6789/0 conn(0x7f531806c380 legacy=0x7f5318143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:40.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 1 monclient: found mon.b 2024-03-25T18:19:40.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:40.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53181456a0 con 0x7f531806d670 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f530dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 5 monclient: authenticate success, global_id 4468 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 10 monclient: _renew_subs 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5318148fc0 con 0x7f531806d670 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 10 monclient: _renew_subs 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5318143aa0 con 0x7f531806d670 2024-03-25T18:19:40.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.379+0000 7f531f64a640 1 librados: init done 2024-03-25T18:19:40.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.380+0000 7f531f64a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:40.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.380+0000 7f531f64a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:40.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.380+0000 7f531f64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:40.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.380+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53180627b0 con 0x7f531806d670 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.662+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5308003610 con 0x7f531806d670 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.662+0000 7f530dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.662+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5308003a60 con 0x7f531806d670 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.662+0000 7f530dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.662+0000 7f530dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.662+0000 7f530dffb640 10 monclient: dump: 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.664 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:40.665 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:40.665 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:40.665 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:40.665 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:40.665 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.662+0000 7f530f7fe640 4 set_mon_vals no callback set 2024-03-25T18:19:40.665 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.663+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f5308003c80 con 0x7f531806d670 2024-03-25T18:19:40.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.664+0000 7f531d3e6640 10 monclient: get_auth_request con 0x7f52fc06df30 auth_method 0 2024-03-25T18:19:40.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.664+0000 7f530dffb640 10 client.4468.objecter ms_handle_connect 0x7f52fc06df30 2024-03-25T18:19:40.669 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.668+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f53080535d0 con 0x7f531806d670 2024-03-25T18:19:40.669 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.668+0000 7f530dffb640 10 client.4468.objecter ms_dispatch 0x7f5318000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:40.669 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.668+0000 7f530dffb640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:40.670 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.668+0000 7f530dffb640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:40.670 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.668+0000 7f530dffb640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T18:19:40.670 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.668+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f530804f0e0 con 0x7f531806d670 2024-03-25T18:19:40.670 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.668+0000 7f530dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:40.670 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.668+0000 7f530dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:40.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.894+0000 7f531f64a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:40.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.894+0000 7f531f64a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:40.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.894+0000 7f531f64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:40.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.894+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5318145830 con 0x7f531806d670 2024-03-25T18:19:40.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.926+0000 7f530dffb640 1 -- 172.21.15.162:0/3429737468 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f5308051010 con 0x7f531806d670 2024-03-25T18:19:40.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.926+0000 7f530dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:40.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.926+0000 7f530dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:40.928 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:19:40.928 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:40.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 10 librados: watch_flush enter 2024-03-25T18:19:40.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 10 librados: watch_flush exit 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 10 client.4468.objecter successfully canceled tick 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 >> v1:172.21.15.191:6800/1552703795 conn(0x7f52fc06df30 legacy=0x7f52fc0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 10 monclient: shutdown 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 >> v1:172.21.15.191:6789/0 conn(0x7f531806d670 legacy=0x7f53181dee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 shutdown_connections 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 >> 172.21.15.162:0/3429737468 conn(0x7f53180b5c00 msgr2=0x7f5318070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 shutdown_connections 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 1 -- 172.21.15.162:0/3429737468 wait complete. 2024-03-25T18:19:40.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:40.928+0000 7f531f64a640 1 librados: shutdown 2024-03-25T18:19:40.942 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:47.749 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:47.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.868+0000 7fd904148640 10 monclient: get_monmap_and_config 2024-03-25T18:19:47.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.868+0000 7fd904148640 10 monclient: build_initial_monmap 2024-03-25T18:19:47.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.868+0000 7fd904148640 10 monclient: monmap: 2024-03-25T18:19:47.869 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:47.869 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:47.869 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:19:47.869462+0000 2024-03-25T18:19:47.869 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:19:47.869462+0000 2024-03-25T18:19:47.869 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:47.870 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:47.870 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:47.870 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:19:47.870 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:19:47.870 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:47.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 1 Processor -- start 2024-03-25T18:19:47.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 1 -- start start 2024-03-25T18:19:47.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 10 monclient: init 2024-03-25T18:19:47.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:47.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:47.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 10 monclient(hunting): picked mon.noname-c con 0x7fd8fc0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:47.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 10 monclient(hunting): picked mon.noname-b con 0x7fd8fc0b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:47.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 10 monclient(hunting): picked mon.noname-a con 0x7fd8fc14be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:47.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0af940 con 0x7fd8fc14be90 2024-03-25T18:19:47.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0b0a70 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.869+0000 7fd904148640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0b0d50 con 0x7fd8fc0b1730 2024-03-25T18:19:47.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.870+0000 7fd904148640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:47.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.870+0000 7fd904148640 10 monclient(hunting): authenticate will time out at 7887.143721s 2024-03-25T18:19:47.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.879+0000 7fd9016e3640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fd8fc0b1730 0x7fd8fc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:41158/0 (socket says 172.21.15.162:41158) 2024-03-25T18:19:47.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.879+0000 7fd9016e3640 1 -- 172.21.15.162:0/2094403024 learned_addr learned my addr 172.21.15.162:0/2094403024 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:47.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.891+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828859758 0 0) 0x7fd8fc0b0d50 con 0x7fd8fc0b1730 2024-03-25T18:19:47.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.891+0000 7fd900ee2640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.891+0000 7fd900ee2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.891+0000 7fd900ee2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.891+0000 7fd900ee2640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:19:47.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.891+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e8003520 con 0x7fd8fc0b1730 2024-03-25T18:19:47.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.893+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026073297 0 0) 0x7fd8fc0b0a70 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.893+0000 7fd900ee2640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.893+0000 7fd900ee2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.893+0000 7fd900ee2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.893+0000 7fd900ee2640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:19:47.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.893+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc0b0d50 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.897+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148574941 0 0) 0x7fd8fc0af940 con 0x7fd8fc14be90 2024-03-25T18:19:47.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.897+0000 7fd900ee2640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.897+0000 7fd900ee2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.897+0000 7fd900ee2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.897+0000 7fd900ee2640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:19:47.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.897+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc0b0a70 con 0x7fd8fc14be90 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566114238 0 0) 0x7fd8e8003520 con 0x7fd8fc0b1730 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 >> v1:172.21.15.191:6789/0 conn(0x7fd8fc0b4d60 legacy=0x7fd8fc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 >> v1:172.21.15.162:6789/0 conn(0x7fd8fc14be90 legacy=0x7fd8fc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 10 monclient: _finish_hunting 0 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 1 monclient: found mon.noname-b 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc0b1030 con 0x7fd8fc0b1730 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 10 monclient: _finish_auth 0 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 10 monclient: _check_auth_tickets 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd8ec001690 con 0x7fd8fc0b1730 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd904148640 5 monclient: authenticate success, global_id 4475 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd904148640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd900ee2640 10 monclient: dump: 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:47.904 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:47.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.901+0000 7fd904148640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:47.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.911+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8ec0017f0 con 0x7fd8fc0b1730 2024-03-25T18:19:47.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.911+0000 7fd900ee2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:47.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.911+0000 7fd900ee2640 1 -- 172.21.15.162:0/2094403024 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd8ec001c00 con 0x7fd8fc0b1730 2024-03-25T18:19:47.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.911+0000 7fd900ee2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.911+0000 7fd900ee2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.911+0000 7fd900ee2640 10 monclient: dump: 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.911+0000 7fd904148640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:47.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.911+0000 7fd904148640 4 set_mon_vals no callback set 2024-03-25T18:19:47.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.912+0000 7fd904148640 10 monclient: shutdown 2024-03-25T18:19:47.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.912+0000 7fd904148640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:47.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.912+0000 7fd904148640 1 -- 172.21.15.162:0/2094403024 >> v1:172.21.15.162:6790/0 conn(0x7fd8fc0b1730 legacy=0x7fd8fc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.912+0000 7fd904148640 1 -- 172.21.15.162:0/2094403024 shutdown_connections 2024-03-25T18:19:47.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.912+0000 7fd904148640 1 -- 172.21.15.162:0/2094403024 >> 172.21.15.162:0/2094403024 conn(0x7fd8fc0617a0 msgr2=0x7fd8fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.912+0000 7fd904148640 1 -- 172.21.15.162:0/2094403024 shutdown_connections 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.912+0000 7fd904148640 1 -- 172.21.15.162:0/2094403024 wait complete. 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 10 monclient: build_initial_monmap 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 10 monclient: monmap: 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:19:47.916 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 1 librados: starting msgr at 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 1 librados: starting objecter 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 1 Processor -- start 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 1 -- start start 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 1 librados: setting wanted keys 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 1 librados: calling monclient init 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.913+0000 7fd904148640 10 monclient: init 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 10 monclient(hunting): picked mon.noname-a con 0x7fd8fc0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 10 monclient(hunting): picked mon.noname-c con 0x7fd8fc0b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 10 monclient(hunting): picked mon.noname-b con 0x7fd8fc14be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc140740 con 0x7fd8fc0b1730 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc140380 con 0x7fd8fc14be90 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc140510 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:47.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.914+0000 7fd904148640 10 monclient(hunting): authenticate will time out at 7887.188041s 2024-03-25T18:19:47.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.932+0000 7fd9026e5640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd8fc14be90 0x7fd8fc1db3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:35918/0 (socket says 172.21.15.162:35918) 2024-03-25T18:19:47.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.932+0000 7fd9026e5640 1 -- 172.21.15.162:0/2273409258 learned_addr learned my addr 172.21.15.162:0/2273409258 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:47.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.934+0000 7fd8f27fc640 10 client.?.objecter ms_handle_connect 0x7fd8fc14be90 2024-03-25T18:19:47.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.934+0000 7fd8f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.936+0000 7fd8f27fc640 10 client.?.objecter ms_handle_connect 0x7fd8fc0b4d60 2024-03-25T18:19:47.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.936+0000 7fd8f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.949+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816815035 0 0) 0x7fd8fc140380 con 0x7fd8fc14be90 2024-03-25T18:19:47.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.949+0000 7fd8f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.949+0000 7fd8f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.949+0000 7fd8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.949+0000 7fd8f27fc640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:19:47.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.949+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8e0004eb0 con 0x7fd8fc14be90 2024-03-25T18:19:47.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.950+0000 7fd8f27fc640 10 client.?.objecter ms_handle_connect 0x7fd8fc0b1730 2024-03-25T18:19:47.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.950+0000 7fd8f27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.956+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220037529 0 0) 0x7fd8fc140510 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.956+0000 7fd8f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.956+0000 7fd8f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.956+0000 7fd8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.956+0000 7fd8f27fc640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:19:47.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.956+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc140380 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.968+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775926674 0 0) 0x7fd8fc140740 con 0x7fd8fc0b1730 2024-03-25T18:19:47.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.968+0000 7fd8f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.968+0000 7fd8f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.968+0000 7fd8f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.968+0000 7fd8f27fc640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:19:47.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.968+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc140510 con 0x7fd8fc0b1730 2024-03-25T18:19:47.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.972+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024855294 0 0) 0x7fd8e0004eb0 con 0x7fd8fc14be90 2024-03-25T18:19:47.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.972+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8fc140740 con 0x7fd8fc14be90 2024-03-25T18:19:47.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.972+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd8f4001c70 con 0x7fd8fc14be90 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.972+0000 7fd8f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.972+0000 7fd8f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.972+0000 7fd8f27fc640 10 monclient(hunting): dump: 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:47.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.972+0000 7fd8f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:47.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.973+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4000461335 0 0) 0x7fd8fc140380 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.973+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8e0004eb0 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.973+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd8ec001b00 con 0x7fd8fc0b4d60 2024-03-25T18:19:47.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.973+0000 7fd8f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.973+0000 7fd8f27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:47.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.973+0000 7fd8f27fc640 10 monclient(hunting): dump: 2024-03-25T18:19:47.975 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:47.975 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:47.976 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.976 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.976 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.976 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:47.976 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:47.976 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:47.976 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:47.976 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:47.985 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.983+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1092539885 0 0) 0x7fd8fc140510 con 0x7fd8fc0b1730 2024-03-25T18:19:47.985 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.983+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8fc140380 con 0x7fd8fc0b1730 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.983+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd8f8001930 con 0x7fd8fc0b1730 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.983+0000 7fd8f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.983+0000 7fd8f27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.983+0000 7fd8f27fc640 10 monclient(hunting): dump: 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:47.986 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46774330 0 0) 0x7fd8fc140740 con 0x7fd8fc14be90 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 >> v1:172.21.15.162:6790/0 conn(0x7fd8fc0b4d60 legacy=0x7fd8fc1daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 >> v1:172.21.15.162:6789/0 conn(0x7fd8fc0b1730 legacy=0x7fd8fc134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 1 monclient: found mon.b 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc137400 con 0x7fd8fc14be90 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd8f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 10 monclient: _renew_subs 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8fc137610 con 0x7fd8fc14be90 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:47.987 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 10 monclient: _renew_subs 2024-03-25T18:19:47.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:47.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc137f80 con 0x7fd8fc14be90 2024-03-25T18:19:47.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.984+0000 7fd904148640 1 librados: init done 2024-03-25T18:19:47.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.985+0000 7fd904148640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.985+0000 7fd904148640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.985+0000 7fd904148640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:47.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:47.985+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc1401f0 con 0x7fd8fc14be90 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.270+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8f4003990 con 0x7fd8fc14be90 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.270+0000 7fd8f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.270+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd8f4003da0 con 0x7fd8fc14be90 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.270+0000 7fd8f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.270+0000 7fd8f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.270+0000 7fd8f27fc640 10 monclient: dump: 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:48.271 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:48.272 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:48.272 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:48.272 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:48.272 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:48.272 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:48.272 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:48.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.270+0000 7fd8f3fff640 4 set_mon_vals no callback set 2024-03-25T18:19:48.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.271+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fd8f4003fc0 con 0x7fd8fc14be90 2024-03-25T18:19:48.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.271+0000 7fd901ee4640 10 monclient: get_auth_request con 0x7fd8e006e0f0 auth_method 0 2024-03-25T18:19:48.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.271+0000 7fd8f27fc640 10 client.4474.objecter ms_handle_connect 0x7fd8e006e0f0 2024-03-25T18:19:48.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.275+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fd8f4054a10 con 0x7fd8fc14be90 2024-03-25T18:19:48.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.275+0000 7fd8f27fc640 10 client.4474.objecter ms_dispatch 0x7fd8fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:48.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.275+0000 7fd8f27fc640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:48.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.275+0000 7fd8f27fc640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:48.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.275+0000 7fd8f27fc640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T18:19:48.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.276+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8f4050520 con 0x7fd8fc14be90 2024-03-25T18:19:48.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.276+0000 7fd8f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:48.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.276+0000 7fd8f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:48.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.534+0000 7fd904148640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:48.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.534+0000 7fd904148640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:48.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.534+0000 7fd904148640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:19:48.536 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.534+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8fc0627b0 con 0x7fd8fc14be90 2024-03-25T18:19:48.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.565+0000 7fd8f27fc640 1 -- 172.21.15.162:0/2273409258 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fd8f4052450 con 0x7fd8fc14be90 2024-03-25T18:19:48.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.565+0000 7fd8f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:48.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.565+0000 7fd8f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:48.566 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:19:48.566 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:48.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 10 librados: watch_flush enter 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 10 librados: watch_flush exit 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 10 client.4474.objecter successfully canceled tick 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 >> v1:172.21.15.191:6800/1552703795 conn(0x7fd8e006e0f0 legacy=0x7fd8e00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 10 monclient: shutdown 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 >> v1:172.21.15.191:6789/0 conn(0x7fd8fc14be90 legacy=0x7fd8fc1db3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 shutdown_connections 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 >> 172.21.15.162:0/2273409258 conn(0x7fd8fc0617a0 msgr2=0x7fd8fc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 shutdown_connections 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 1 -- 172.21.15.162:0/2273409258 wait complete. 2024-03-25T18:19:48.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:48.568+0000 7fd904148640 1 librados: shutdown 2024-03-25T18:19:48.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:52.922 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:05.149 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.929 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:05.157 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.965 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:05.193 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.979 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:05.195 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:52.982 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:05.199 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:53.003 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:05.231 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:53.011 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:05.238 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:53.013 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:05.229 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:53.035 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:05.263 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.387 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:55.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.503+0000 7f41229a8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:55.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.503+0000 7f41229a8640 10 monclient: build_initial_monmap 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.503+0000 7f41229a8640 10 monclient: monmap: 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:19:55.504400+0000 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:19:55.504400+0000 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:19:55.505 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:55.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.504+0000 7f41229a8640 1 Processor -- start 2024-03-25T18:19:55.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.504+0000 7f41229a8640 1 -- start start 2024-03-25T18:19:55.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.504+0000 7f41229a8640 10 monclient: init 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 10 monclient(hunting): picked mon.noname-c con 0x7f411c0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 10 monclient(hunting): picked mon.noname-b con 0x7f411c0b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 10 monclient(hunting): picked mon.noname-a con 0x7f411c14be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0af940 con 0x7f411c14be90 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0b0a70 con 0x7f411c0b4d60 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0b0d50 con 0x7f411c0b1730 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:55.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.505+0000 7f41229a8640 10 monclient(hunting): authenticate will time out at 7894.778978s 2024-03-25T18:19:55.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.514+0000 7f411bfff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f411c0b4d60 0x7f411c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:33766/0 (socket says 172.21.15.162:33766) 2024-03-25T18:19:55.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.514+0000 7f411bfff640 1 -- 172.21.15.162:0/1703668046 learned_addr learned my addr 172.21.15.162:0/1703668046 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:55.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.529+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101219883 0 0) 0x7f411c0b0a70 con 0x7f411c0b4d60 2024-03-25T18:19:55.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.529+0000 7f411affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:55.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.529+0000 7f411affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:55.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.529+0000 7f411affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:55.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.529+0000 7f411affd640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:19:55.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.529+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4104003520 con 0x7f411c0b4d60 2024-03-25T18:19:55.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.532+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834789818 0 0) 0x7f411c0af940 con 0x7f411c14be90 2024-03-25T18:19:55.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.532+0000 7f411affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:55.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.532+0000 7f411affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:55.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.532+0000 7f411affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:55.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.532+0000 7f411affd640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:19:55.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.532+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c0b0a70 con 0x7f411c14be90 2024-03-25T18:19:55.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1838549071 0 0) 0x7f411c0b0a70 con 0x7f411c14be90 2024-03-25T18:19:55.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 >> v1:172.21.15.162:6790/0 conn(0x7f411c0b1730 legacy=0x7f411c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 >> v1:172.21.15.191:6789/0 conn(0x7f411c0b4d60 legacy=0x7f411c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 1 monclient: found mon.noname-a 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c0b1030 con 0x7f411c14be90 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 10 monclient: _finish_auth 0 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f41140019f0 con 0x7f411c14be90 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f411affd640 10 monclient: dump: 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.546 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.547 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:55.547 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:55.547 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:55.547 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:55.547 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:55.547 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:55.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f41229a8640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:19:55.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.544+0000 7f41229a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4114001b50 con 0x7f411c14be90 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f411affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f411affd640 1 -- 172.21.15.162:0/1703668046 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4114001f20 con 0x7f411c14be90 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f411affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f411affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f411affd640 10 monclient: dump: 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:55.553 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:55.554 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:55.554 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:55.554 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:55.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f41229a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:55.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f41229a8640 4 set_mon_vals no callback set 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f41229a8640 10 monclient: shutdown 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f41229a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f41229a8640 1 -- 172.21.15.162:0/1703668046 >> v1:172.21.15.162:6789/0 conn(0x7f411c14be90 legacy=0x7f411c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f41229a8640 1 -- 172.21.15.162:0/1703668046 shutdown_connections 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.552+0000 7f41229a8640 1 -- 172.21.15.162:0/1703668046 >> 172.21.15.162:0/1703668046 conn(0x7f411c0617a0 msgr2=0x7f411c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.553+0000 7f41229a8640 1 -- 172.21.15.162:0/1703668046 shutdown_connections 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.553+0000 7f41229a8640 1 -- 172.21.15.162:0/1703668046 wait complete. 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.553+0000 7f41229a8640 10 monclient: build_initial_monmap 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.553+0000 7f41229a8640 10 monclient: monmap: 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:19:55.556 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.553+0000 7f41229a8640 1 librados: starting msgr at 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.553+0000 7f41229a8640 1 librados: starting objecter 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 1 Processor -- start 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 1 -- start start 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 1 librados: setting wanted keys 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 1 librados: calling monclient init 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 10 monclient: init 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:55.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:55.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 10 monclient(hunting): picked mon.noname-a con 0x7f411c0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:19:55.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 10 monclient(hunting): picked mon.noname-c con 0x7f411c0b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:19:55.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 10 monclient(hunting): picked mon.noname-b con 0x7f411c14be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:19:55.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c13bfd0 con 0x7f411c0b1730 2024-03-25T18:19:55.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c1d7af0 con 0x7f411c14be90 2024-03-25T18:19:55.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c13a580 con 0x7f411c0b4d60 2024-03-25T18:19:55.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:55.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.554+0000 7f41229a8640 10 monclient(hunting): authenticate will time out at 7894.828586s 2024-03-25T18:19:55.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.570+0000 7f411b7fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f411c0b4d60 0x7f411c1d7f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:33884/0 (socket says 172.21.15.162:33884) 2024-03-25T18:19:55.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.570+0000 7f411b7fe640 1 -- 172.21.15.162:0/3268417362 learned_addr learned my addr 172.21.15.162:0/3268417362 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:19:55.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.571+0000 7f4118ff9640 10 client.?.objecter ms_handle_connect 0x7f411c0b4d60 2024-03-25T18:19:55.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.571+0000 7f4118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:55.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.572+0000 7f4118ff9640 10 client.?.objecter ms_handle_connect 0x7f411c14be90 2024-03-25T18:19:55.573 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.572+0000 7f4118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:55.574 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.573+0000 7f4118ff9640 10 client.?.objecter ms_handle_connect 0x7f411c0b1730 2024-03-25T18:19:55.574 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.573+0000 7f4118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:55.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.578+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507789207 0 0) 0x7f411c13a580 con 0x7f411c0b4d60 2024-03-25T18:19:55.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.578+0000 7f4118ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:55.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.578+0000 7f4118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:55.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.578+0000 7f4118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:55.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.578+0000 7f4118ff9640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:19:55.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.578+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4108004eb0 con 0x7f411c0b4d60 2024-03-25T18:19:55.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658015593 0 0) 0x7f411c13bfd0 con 0x7f411c0b1730 2024-03-25T18:19:55.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:55.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:55.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:55.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:19:55.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c13a580 con 0x7f411c0b1730 2024-03-25T18:19:55.587 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252326844 0 0) 0x7f411c1d7af0 con 0x7f411c14be90 2024-03-25T18:19:55.587 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:55.587 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:55.587 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:55.587 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:19:55.587 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.585+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c13bfd0 con 0x7f411c14be90 2024-03-25T18:19:55.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.600+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1659872104 0 0) 0x7f4108004eb0 con 0x7f411c0b4d60 2024-03-25T18:19:55.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.600+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f411c1d7af0 con 0x7f411c0b4d60 2024-03-25T18:19:55.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.600+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f41100019d0 con 0x7f411c0b4d60 2024-03-25T18:19:55.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.600+0000 7f4118ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:55.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.600+0000 7f4118ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:55.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.600+0000 7f4118ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:55.602 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.600+0000 7f4118ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.601+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505952926 0 0) 0x7f411c13bfd0 con 0x7f411c14be90 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.601+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4108004eb0 con 0x7f411c14be90 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.601+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4114001e60 con 0x7f411c14be90 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.601+0000 7f4118ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.601+0000 7f4118ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:55.603 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.601+0000 7f4118ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:55.604 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:55.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2890610400 0 0) 0x7f411c1d7af0 con 0x7f411c0b4d60 2024-03-25T18:19:55.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 >> v1:172.21.15.191:6789/0 conn(0x7f411c14be90 legacy=0x7f411c1370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 >> v1:172.21.15.162:6789/0 conn(0x7f411c0b1730 legacy=0x7f411c13af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:55.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 1 monclient: found mon.c 2024-03-25T18:19:55.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:55.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c13a860 con 0x7f411c0b4d60 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f4118ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 5 monclient: authenticate success, global_id 4484 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 10 monclient: _renew_subs 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c1d7730 con 0x7f411c0b4d60 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 10 monclient: _renew_subs 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f411c1379b0 con 0x7f411c0b4d60 2024-03-25T18:19:55.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.614+0000 7f41229a8640 1 librados: init done 2024-03-25T18:19:55.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.615+0000 7f41229a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:55.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.615+0000 7f41229a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:55.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.615+0000 7f41229a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:55.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.615+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411c0627b0 con 0x7f411c0b4d60 2024-03-25T18:19:55.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.938+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4110003610 con 0x7f411c0b4d60 2024-03-25T18:19:55.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.938+0000 7f4118ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:55.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.938+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f41100039a0 con 0x7f411c0b4d60 2024-03-25T18:19:55.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.938+0000 7f4118ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:55.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.938+0000 7f4118ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:55.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.938+0000 7f4118ff9640 10 monclient: dump: 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:19:55.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.938+0000 7f411a7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:55.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.939+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f4110003bc0 con 0x7f411c0b4d60 2024-03-25T18:19:55.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.939+0000 7f411bfff640 10 monclient: get_auth_request con 0x7f410806df70 auth_method 0 2024-03-25T18:19:55.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.940+0000 7f4118ff9640 10 client.4484.objecter ms_handle_connect 0x7f410806df70 2024-03-25T18:19:55.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.945+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f4110054810 con 0x7f411c0b4d60 2024-03-25T18:19:55.947 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.945+0000 7f4118ff9640 10 client.4484.objecter ms_dispatch 0x7f411c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:55.947 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.945+0000 7f4118ff9640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:55.947 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.945+0000 7f4118ff9640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:55.947 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.945+0000 7f4118ff9640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-25T18:19:55.947 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.945+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4110050320 con 0x7f411c0b4d60 2024-03-25T18:19:55.947 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.945+0000 7f4118ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:55.947 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:55.945+0000 7f4118ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:56.198 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.197+0000 7f41229a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:56.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.197+0000 7f41229a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:56.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.197+0000 7f41229a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:19:56.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.197+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f411c1d7ca0 con 0x7f411c0b4d60 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.225+0000 7f4118ff9640 1 -- 172.21.15.162:0/3268417362 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f411c1d7ca0 con 0x7f411c0b4d60 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.225+0000 7f4118ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.225+0000 7f4118ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:19:56.226 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 10 librados: watch_flush enter 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 10 librados: watch_flush exit 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 10 client.4484.objecter successfully canceled tick 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 >> v1:172.21.15.191:6800/1552703795 conn(0x7f410806df70 legacy=0x7f4108070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 10 monclient: shutdown 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 >> v1:172.21.15.162:6790/0 conn(0x7f411c0b4d60 legacy=0x7f411c1d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 shutdown_connections 2024-03-25T18:19:56.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.227+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 >> 172.21.15.162:0/3268417362 conn(0x7f411c0617a0 msgr2=0x7f411c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:56.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.228+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 shutdown_connections 2024-03-25T18:19:56.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.228+0000 7f41229a8640 1 -- 172.21.15.162:0/3268417362 wait complete. 2024-03-25T18:19:56.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:19:56.228+0000 7f41229a8640 1 librados: shutdown 2024-03-25T18:19:56.241 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:03.047 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:03.161 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.160+0000 7f7ff7d7c640 10 monclient: get_monmap_and_config 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.160+0000 7f7ff7d7c640 10 monclient: build_initial_monmap 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.160+0000 7f7ff7d7c640 10 monclient: monmap: 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:20:03.161952+0000 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:20:03.161952+0000 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:20:03.162 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.161+0000 7f7ff7d7c640 1 Processor -- start 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.161+0000 7f7ff7d7c640 1 -- start start 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.161+0000 7f7ff7d7c640 10 monclient: init 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff00b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff00b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff014be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff00af940 con 0x7f7ff014be90 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff00b0a70 con 0x7f7ff00b1730 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff00b0d50 con 0x7f7ff00b4d60 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:03.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.162+0000 7f7ff7d7c640 10 monclient(hunting): authenticate will time out at 7902.436341s 2024-03-25T18:20:03.172 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.171+0000 7f7ff6319640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f7ff014be90 0x7f7ff014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:48002/0 (socket says 172.21.15.162:48002) 2024-03-25T18:20:03.172 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.171+0000 7f7ff6319640 1 -- 172.21.15.162:0/2009918722 learned_addr learned my addr 172.21.15.162:0/2009918722 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:03.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.191+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043125807 0 0) 0x7f7ff00b0d50 con 0x7f7ff00b4d60 2024-03-25T18:20:03.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.191+0000 7f7ff4b16640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.191+0000 7f7ff4b16640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.191+0000 7f7ff4b16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.191+0000 7f7ff4b16640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:20:03.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.191+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc003520 con 0x7f7ff00b4d60 2024-03-25T18:20:03.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.194+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003686822 0 0) 0x7f7ff00af940 con 0x7f7ff014be90 2024-03-25T18:20:03.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.194+0000 7f7ff4b16640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.194+0000 7f7ff4b16640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.194+0000 7f7ff4b16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.194+0000 7f7ff4b16640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:20:03.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.194+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff00b0d50 con 0x7f7ff014be90 2024-03-25T18:20:03.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.198+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961455036 0 0) 0x7f7ff00b0a70 con 0x7f7ff00b1730 2024-03-25T18:20:03.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.198+0000 7f7ff4b16640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.199 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.198+0000 7f7ff4b16640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.198+0000 7f7ff4b16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.198+0000 7f7ff4b16640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:20:03.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.198+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff00af940 con 0x7f7ff00b1730 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4123520625 0 0) 0x7f7ff00af940 con 0x7f7ff00b1730 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 >> v1:172.21.15.162:6790/0 conn(0x7f7ff00b4d60 legacy=0x7f7ff013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 >> v1:172.21.15.162:6789/0 conn(0x7f7ff014be90 legacy=0x7f7ff014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 10 monclient: _finish_hunting 0 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 1 monclient: found mon.noname-c 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff00b1030 con 0x7f7ff00b1730 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 10 monclient: _finish_auth 0 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 10 monclient: _check_auth_tickets 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:03.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7fe00015b0 con 0x7f7ff00b1730 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff7d7c640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff7d7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff4b16640 10 monclient: dump: 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:03.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.217+0000 7f7ff7d7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.240+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fe0001710 con 0x7f7ff00b1730 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.240+0000 7f7ff4b16640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.240+0000 7f7ff4b16640 1 -- 172.21.15.162:0/2009918722 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7fe0001b60 con 0x7f7ff00b1730 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.240+0000 7f7ff7d7c640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.240+0000 7f7ff7d7c640 4 set_mon_vals no callback set 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.240+0000 7f7ff4b16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.240+0000 7f7ff4b16640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.240+0000 7f7ff4b16640 10 monclient: dump: 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:03.241 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:03.242 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:03.242 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:03.242 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:03.242 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:03.242 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:03.242 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:03.242 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:03.242 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:03.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.241+0000 7f7ff7d7c640 10 monclient: shutdown 2024-03-25T18:20:03.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.241+0000 7f7ff7d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:03.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.241+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/2009918722 >> v1:172.21.15.191:6789/0 conn(0x7f7ff00b1730 legacy=0x7f7ff00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.241+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/2009918722 shutdown_connections 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.241+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/2009918722 >> 172.21.15.162:0/2009918722 conn(0x7f7ff00617a0 msgr2=0x7f7ff0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.241+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/2009918722 shutdown_connections 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.241+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/2009918722 wait complete. 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 10 monclient: build_initial_monmap 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 10 monclient: monmap: 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:03.244 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 1 librados: starting msgr at 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 1 librados: starting objecter 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 1 Processor -- start 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 1 -- start start 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 1 librados: setting wanted keys 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 1 librados: calling monclient init 2024-03-25T18:20:03.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.242+0000 7f7ff7d7c640 10 monclient: init 2024-03-25T18:20:03.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:03.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:03.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff00b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:03.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff00b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:03.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff014be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:03.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff0140a50 con 0x7f7ff014be90 2024-03-25T18:20:03.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff0137aa0 con 0x7f7ff00b4d60 2024-03-25T18:20:03.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff0137c30 con 0x7f7ff00b1730 2024-03-25T18:20:03.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:03.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.243+0000 7f7ff7d7c640 10 monclient(hunting): authenticate will time out at 7902.517200s 2024-03-25T18:20:03.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.258+0000 7f7ff5b18640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f7ff00b1730 0x7f7ff0131d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:38810/0 (socket says 172.21.15.162:38810) 2024-03-25T18:20:03.259 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.258+0000 7f7ff5b18640 1 -- 172.21.15.162:0/1647193185 learned_addr learned my addr 172.21.15.162:0/1647193185 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:03.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.259+0000 7f7fe67fc640 10 client.?.objecter ms_handle_connect 0x7f7ff00b1730 2024-03-25T18:20:03.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.259+0000 7f7fe67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:03.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.268+0000 7f7fe67fc640 10 client.?.objecter ms_handle_connect 0x7f7ff014be90 2024-03-25T18:20:03.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.268+0000 7f7fe67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:03.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.272+0000 7f7fe67fc640 10 client.?.objecter ms_handle_connect 0x7f7ff00b4d60 2024-03-25T18:20:03.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.272+0000 7f7fe67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:03.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.274+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549226185 0 0) 0x7f7ff0137c30 con 0x7f7ff00b1730 2024-03-25T18:20:03.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.274+0000 7f7fe67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.274+0000 7f7fe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.274+0000 7f7fe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.274+0000 7f7fe67fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:20:03.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.274+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fd4004eb0 con 0x7f7ff00b1730 2024-03-25T18:20:03.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.282+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751282036 0 0) 0x7f7ff0140a50 con 0x7f7ff014be90 2024-03-25T18:20:03.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.282+0000 7f7fe67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.282+0000 7f7fe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.282+0000 7f7fe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.282+0000 7f7fe67fc640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:20:03.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.282+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff0137c30 con 0x7f7ff014be90 2024-03-25T18:20:03.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.288+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156962163 0 0) 0x7f7ff0137aa0 con 0x7f7ff00b4d60 2024-03-25T18:20:03.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.289+0000 7f7fe67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:03.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.289+0000 7f7fe67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:03.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.289+0000 7f7fe67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:03.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.289+0000 7f7fe67fc640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:20:03.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.289+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff0140a50 con 0x7f7ff00b4d60 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.294+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3627469648 0 0) 0x7f7fd4004eb0 con 0x7f7ff00b1730 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.294+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff0137aa0 con 0x7f7ff00b1730 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.294+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7fec001990 con 0x7f7ff00b1730 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.294+0000 7f7fe67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.294+0000 7f7fe67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.294+0000 7f7fe67fc640 10 monclient(hunting): dump: 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:03.295 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:03.296 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:03.296 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:03.296 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:03.296 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:03.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.294+0000 7f7fe67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:03.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 957758203 0 0) 0x7f7ff0137aa0 con 0x7f7ff00b1730 2024-03-25T18:20:03.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 >> v1:172.21.15.191:6789/0 conn(0x7f7ff00b4d60 legacy=0x7f7ff01423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 >> v1:172.21.15.162:6789/0 conn(0x7f7ff014be90 legacy=0x7f7ff0142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 1 monclient: found mon.c 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff0137dc0 con 0x7f7ff00b1730 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7fe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 5 monclient: authenticate success, global_id 4490 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 10 monclient: _renew_subs 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff0144d40 con 0x7f7ff00b1730 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 10 monclient: _renew_subs 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:03.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff0132490 con 0x7f7ff00b1730 2024-03-25T18:20:03.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.309+0000 7f7ff7d7c640 1 librados: init done 2024-03-25T18:20:03.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.310+0000 7f7ff7d7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:03.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.310+0000 7f7ff7d7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:03.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.310+0000 7f7ff7d7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:03.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.310+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff00627b0 con 0x7f7ff00b1730 2024-03-25T18:20:03.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.616+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fec0035f0 con 0x7f7ff00b1730 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.616+0000 7f7fe67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.616+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7fec003980 con 0x7f7ff00b1730 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.616+0000 7f7fe67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.616+0000 7f7fe67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.616+0000 7f7fe67fc640 10 monclient: dump: 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:03.618 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:03.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.616+0000 7f7fe7fff640 4 set_mon_vals no callback set 2024-03-25T18:20:03.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.617+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f7fec003ba0 con 0x7f7ff00b1730 2024-03-25T18:20:03.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.617+0000 7f7ff5317640 10 monclient: get_auth_request con 0x7f7fd406ddf0 auth_method 0 2024-03-25T18:20:03.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.618+0000 7f7fe67fc640 10 client.4490.objecter ms_handle_connect 0x7f7fd406ddf0 2024-03-25T18:20:03.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.626+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f7fec054830 con 0x7f7ff00b1730 2024-03-25T18:20:03.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.626+0000 7f7fe67fc640 10 client.4490.objecter ms_dispatch 0x7f7ff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:03.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.626+0000 7f7fe67fc640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:03.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.626+0000 7f7fe67fc640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:03.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.627+0000 7f7fe67fc640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-25T18:20:03.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.627+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fec050340 con 0x7f7ff00b1730 2024-03-25T18:20:03.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.627+0000 7f7fe67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:03.628 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.627+0000 7f7fe67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:03.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.847+0000 7f7ff7d7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:03.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.847+0000 7f7ff7d7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:03.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.847+0000 7f7ff7d7c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:03.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.847+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ff0140be0 con 0x7f7ff00b1730 2024-03-25T18:20:03.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.884+0000 7f7fe67fc640 1 -- 172.21.15.162:0/1647193185 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f7fec052270 con 0x7f7ff00b1730 2024-03-25T18:20:03.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.884+0000 7f7fe67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:03.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.884+0000 7f7fe67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:03.885 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:20:03.885 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:03.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 10 librados: watch_flush enter 2024-03-25T18:20:03.887 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 10 librados: watch_flush exit 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 10 client.4490.objecter successfully canceled tick 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 >> v1:172.21.15.191:6800/1552703795 conn(0x7f7fd406ddf0 legacy=0x7f7fd40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 10 monclient: shutdown 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 >> v1:172.21.15.162:6790/0 conn(0x7f7ff00b1730 legacy=0x7f7ff0131d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 shutdown_connections 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 >> 172.21.15.162:0/1647193185 conn(0x7f7ff00617a0 msgr2=0x7f7ff00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 shutdown_connections 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 1 -- 172.21.15.162:0/1647193185 wait complete. 2024-03-25T18:20:03.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:03.886+0000 7f7ff7d7c640 1 librados: shutdown 2024-03-25T18:20:03.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:10.705 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.819+0000 7f391fc82640 10 monclient: get_monmap_and_config 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.819+0000 7f391fc82640 10 monclient: build_initial_monmap 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.819+0000 7f391fc82640 10 monclient: monmap: 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:20:10.820353+0000 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:20:10.820353+0000 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:10.820 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:10.821 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:10.821 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:20:10.821 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:20:10.821 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.819+0000 7f391fc82640 1 Processor -- start 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 1 -- start start 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 10 monclient: init 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 10 monclient(hunting): picked mon.noname-b con 0x7f39180b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 10 monclient(hunting): picked mon.noname-a con 0x7f39180b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 10 monclient(hunting): picked mon.noname-c con 0x7f391814be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:10.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39180af940 con 0x7f39180b1730 2024-03-25T18:20:10.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39180b0a70 con 0x7f391814be90 2024-03-25T18:20:10.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39180b0d50 con 0x7f39180b4d60 2024-03-25T18:20:10.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:10.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.820+0000 7f391fc82640 10 monclient(hunting): authenticate will time out at 7910.094573s 2024-03-25T18:20:10.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.822+0000 7f391e21f640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f391814be90 0x7f391814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:58044/0 (socket says 172.21.15.162:58044) 2024-03-25T18:20:10.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.822+0000 7f391e21f640 1 -- 172.21.15.162:0/1444082617 learned_addr learned my addr 172.21.15.162:0/1444082617 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:10.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.912+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221134197 0 0) 0x7f39180af940 con 0x7f39180b1730 2024-03-25T18:20:10.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.912+0000 7f391ca1c640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.912+0000 7f391ca1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.913 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.912+0000 7f391ca1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.912+0000 7f391ca1c640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:20:10.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.912+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3904003520 con 0x7f39180b1730 2024-03-25T18:20:10.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.918+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883807418 0 0) 0x7f39180b0a70 con 0x7f391814be90 2024-03-25T18:20:10.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.918+0000 7f391ca1c640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.918+0000 7f391ca1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.918+0000 7f391ca1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.918+0000 7f391ca1c640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:20:10.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.918+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39180af940 con 0x7f391814be90 2024-03-25T18:20:10.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.920+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868604130 0 0) 0x7f39180b0d50 con 0x7f39180b4d60 2024-03-25T18:20:10.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.920+0000 7f391ca1c640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.920+0000 7f391ca1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.920+0000 7f391ca1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.920+0000 7f391ca1c640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:20:10.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.920+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39180b0a70 con 0x7f39180b4d60 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3805137858 0 0) 0x7f39180af940 con 0x7f391814be90 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 >> v1:172.21.15.162:6790/0 conn(0x7f39180b4d60 legacy=0x7f391813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 >> v1:172.21.15.162:6789/0 conn(0x7f39180b1730 legacy=0x7f39180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 10 monclient: _finish_hunting 0 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 1 monclient: found mon.noname-c 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39180b1030 con 0x7f391814be90 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 10 monclient: _finish_auth 0 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 10 monclient: _check_auth_tickets 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:10.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f39100019f0 con 0x7f391814be90 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391ca1c640 10 monclient: dump: 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391fc82640 5 monclient: authenticate success, global_id 4489 2024-03-25T18:20:10.932 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.929+0000 7f391fc82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.939+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3910001c10 con 0x7f391814be90 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.939+0000 7f391ca1c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.939+0000 7f391ca1c640 1 -- 172.21.15.162:0/1444082617 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3910001fa0 con 0x7f391814be90 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.939+0000 7f391fc82640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.939+0000 7f391fc82640 4 set_mon_vals no callback set 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.939+0000 7f391ca1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.939+0000 7f391ca1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.939+0000 7f391ca1c640 10 monclient: dump: 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:10.940 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:10.941 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:10.941 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.941 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:10.941 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:10.941 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:10.941 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:10.941 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:10.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.940+0000 7f391fc82640 10 monclient: shutdown 2024-03-25T18:20:10.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.940+0000 7f391fc82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:10.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.940+0000 7f391fc82640 1 -- 172.21.15.162:0/1444082617 >> v1:172.21.15.191:6789/0 conn(0x7f391814be90 legacy=0x7f391814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.940+0000 7f391fc82640 1 -- 172.21.15.162:0/1444082617 shutdown_connections 2024-03-25T18:20:10.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.940+0000 7f391fc82640 1 -- 172.21.15.162:0/1444082617 >> 172.21.15.162:0/1444082617 conn(0x7f39180617a0 msgr2=0x7f3918061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:10.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.940+0000 7f391fc82640 1 -- 172.21.15.162:0/1444082617 shutdown_connections 2024-03-25T18:20:10.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.940+0000 7f391fc82640 1 -- 172.21.15.162:0/1444082617 wait complete. 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 10 monclient: build_initial_monmap 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 10 monclient: monmap: 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 1 librados: starting msgr at 2024-03-25T18:20:10.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 1 librados: starting objecter 2024-03-25T18:20:10.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 1 Processor -- start 2024-03-25T18:20:10.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 1 -- start start 2024-03-25T18:20:10.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 1 librados: setting wanted keys 2024-03-25T18:20:10.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 1 librados: calling monclient init 2024-03-25T18:20:10.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.941+0000 7f391fc82640 10 monclient: init 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.942+0000 7f391fc82640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.942+0000 7f391fc82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.942+0000 7f391fc82640 10 monclient(hunting): picked mon.noname-b con 0x7f39180b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.942+0000 7f391fc82640 10 monclient(hunting): picked mon.noname-a con 0x7f39180b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.943+0000 7f391fc82640 10 monclient(hunting): picked mon.noname-c con 0x7f391814be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.943+0000 7f391fc82640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3918142cc0 con 0x7f39180b4d60 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.943+0000 7f391fc82640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3918142fa0 con 0x7f39180b1730 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.943+0000 7f391fc82640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39181db6a0 con 0x7f391814be90 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.943+0000 7f391fc82640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:10.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.943+0000 7f391fc82640 10 monclient(hunting): authenticate will time out at 7910.216778s 2024-03-25T18:20:10.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.945+0000 7f391da1e640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f39180b1730 0x7f39181dacb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:58106/0 (socket says 172.21.15.162:58106) 2024-03-25T18:20:10.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.945+0000 7f391da1e640 1 -- 172.21.15.162:0/1252672581 learned_addr learned my addr 172.21.15.162:0/1252672581 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:10.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.945+0000 7f390e7fc640 10 client.?.objecter ms_handle_connect 0x7f39180b1730 2024-03-25T18:20:10.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.945+0000 7f390e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.960+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668008775 0 0) 0x7f3918142fa0 con 0x7f39180b1730 2024-03-25T18:20:10.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.960+0000 7f390e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.960+0000 7f390e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.961 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.960+0000 7f390e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.960+0000 7f390e7fc640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:20:10.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.960+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38fc004eb0 con 0x7f39180b1730 2024-03-25T18:20:10.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.962+0000 7f390e7fc640 10 client.?.objecter ms_handle_connect 0x7f39180b4d60 2024-03-25T18:20:10.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.962+0000 7f390e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.966+0000 7f390e7fc640 10 client.?.objecter ms_handle_connect 0x7f391814be90 2024-03-25T18:20:10.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.966+0000 7f390e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.973+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4111401383 0 0) 0x7f38fc004eb0 con 0x7f39180b1730 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.973+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3918142fa0 con 0x7f39180b1730 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.973+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3914001a10 con 0x7f39180b1730 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.973+0000 7f390e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.973+0000 7f390e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.973+0000 7f390e7fc640 10 monclient(hunting): dump: 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:10.975 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:10.976 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:10.976 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:10.976 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.973+0000 7f390e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:10.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.976+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267970876 0 0) 0x7f3918142cc0 con 0x7f39180b4d60 2024-03-25T18:20:10.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.976+0000 7f390e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.976+0000 7f390e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.976+0000 7f390e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.976+0000 7f390e7fc640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:20:10.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.976+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38fc004eb0 con 0x7f39180b4d60 2024-03-25T18:20:10.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.980+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204616600 0 0) 0x7f39181db6a0 con 0x7f391814be90 2024-03-25T18:20:10.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.980+0000 7f390e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.980+0000 7f390e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.980+0000 7f390e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.980+0000 7f390e7fc640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:20:10.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.980+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3918142cc0 con 0x7f391814be90 2024-03-25T18:20:10.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1130506136 0 0) 0x7f3918142fa0 con 0x7f39180b1730 2024-03-25T18:20:10.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 >> v1:172.21.15.162:6790/0 conn(0x7f391814be90 legacy=0x7f3918141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 >> v1:172.21.15.162:6789/0 conn(0x7f39180b4d60 legacy=0x7f3918132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 1 monclient: found mon.b 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39181db3c0 con 0x7f39180b1730 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f390e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 5 monclient: authenticate success, global_id 4492 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 10 monclient: _renew_subs 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39181d5000 con 0x7f39180b1730 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 10 client.4492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:10.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 10 monclient: _renew_subs 2024-03-25T18:20:10.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:10.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3918133680 con 0x7f39180b1730 2024-03-25T18:20:10.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.987+0000 7f391fc82640 1 librados: init done 2024-03-25T18:20:10.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.988+0000 7f391fc82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.988+0000 7f391fc82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.988+0000 7f391fc82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:10.990 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:10.988+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39181db830 con 0x7f39180b1730 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.267+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3914001cb0 con 0x7f39180b1730 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.267+0000 7f390e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.267+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3914003a60 con 0x7f39180b1730 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.267+0000 7f390e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.267+0000 7f390e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.267+0000 7f390e7fc640 10 monclient: dump: 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:11.269 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:11.270 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:11.270 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:11.270 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:11.270 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:11.270 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:11.270 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:11.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.267+0000 7f390ffff640 4 set_mon_vals no callback set 2024-03-25T18:20:11.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.269+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f3914003c80 con 0x7f39180b1730 2024-03-25T18:20:11.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.269+0000 7f391d21d640 10 monclient: get_auth_request con 0x7f38fc06de80 auth_method 0 2024-03-25T18:20:11.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.270+0000 7f390e7fc640 10 client.4492.objecter ms_handle_connect 0x7f38fc06de80 2024-03-25T18:20:11.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.275+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f39140548a0 con 0x7f39180b1730 2024-03-25T18:20:11.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.275+0000 7f390e7fc640 10 client.4492.objecter ms_dispatch 0x7f3918000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:11.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.275+0000 7f390e7fc640 3 client.4492.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:11.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.275+0000 7f390e7fc640 3 client.4492.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:11.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.275+0000 7f390e7fc640 20 client.4492.objecter dump_active .. 0 homeless 2024-03-25T18:20:11.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.275+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39140503b0 con 0x7f39180b1730 2024-03-25T18:20:11.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.275+0000 7f390e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:11.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.275+0000 7f390e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:11.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.534+0000 7f391fc82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:11.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.534+0000 7f391fc82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:11.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.534+0000 7f391fc82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:11.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.534+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39180627b0 con 0x7f39180b1730 2024-03-25T18:20:11.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.556+0000 7f390e7fc640 1 -- 172.21.15.162:0/1252672581 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f39140522e0 con 0x7f39180b1730 2024-03-25T18:20:11.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.556+0000 7f390e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:11.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.556+0000 7f390e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:11.557 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:20:11.558 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:11.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 10 librados: watch_flush enter 2024-03-25T18:20:11.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 10 librados: watch_flush exit 2024-03-25T18:20:11.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 20 client.4492.objecter shutdown clearing up homeless session... 2024-03-25T18:20:11.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 10 client.4492.objecter successfully canceled tick 2024-03-25T18:20:11.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 >> v1:172.21.15.191:6800/1552703795 conn(0x7f38fc06de80 legacy=0x7f38fc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:11.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 10 monclient: shutdown 2024-03-25T18:20:11.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:11.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 >> v1:172.21.15.191:6789/0 conn(0x7f39180b1730 legacy=0x7f39181dacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:11.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 shutdown_connections 2024-03-25T18:20:11.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 >> 172.21.15.162:0/1252672581 conn(0x7f39180617a0 msgr2=0x7f39180b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:11.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 shutdown_connections 2024-03-25T18:20:11.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 1 -- 172.21.15.162:0/1252672581 wait complete. 2024-03-25T18:20:11.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:11.559+0000 7f391fc82640 1 librados: shutdown 2024-03-25T18:20:11.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:17.986 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:30.214 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:17.995 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:30.223 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:18.037 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:30.264 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:18.048 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:30.263 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:18.053 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:30.269 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:18.059 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:30.287 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:18.072 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:30.300 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:18.078 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:30.294 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:18.112 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:30.340 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:18.380 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:18.495 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.494+0000 7fa99ec0f640 10 monclient: get_monmap_and_config 2024-03-25T18:20:18.495 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.494+0000 7fa99ec0f640 10 monclient: build_initial_monmap 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.494+0000 7fa99ec0f640 10 monclient: monmap: 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:20:18.495550+0000 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:20:18.495550+0000 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:20:18.496 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:18.498 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.495+0000 7fa99ec0f640 1 Processor -- start 2024-03-25T18:20:18.498 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.495+0000 7fa99ec0f640 1 -- start start 2024-03-25T18:20:18.498 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.495+0000 7fa99ec0f640 10 monclient: init 2024-03-25T18:20:18.498 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:18.498 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:18.498 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 10 monclient(hunting): picked mon.noname-c con 0x7fa9980a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:18.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 10 monclient(hunting): picked mon.noname-a con 0x7fa99813f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:18.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 10 monclient(hunting): picked mon.noname-b con 0x7fa9981430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:18.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa998142520 con 0x7fa99813f710 2024-03-25T18:20:18.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980b28a0 con 0x7fa9980a98f0 2024-03-25T18:20:18.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980b2a30 con 0x7fa9981430c0 2024-03-25T18:20:18.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:18.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.496+0000 7fa99ec0f640 10 monclient(hunting): authenticate will time out at 7917.770200s 2024-03-25T18:20:18.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.502+0000 7fa99d1ac640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fa9981430c0 0x7fa99814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:60342/0 (socket says 172.21.15.162:60342) 2024-03-25T18:20:18.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.502+0000 7fa99d1ac640 1 -- 172.21.15.162:0/2376432314 learned_addr learned my addr 172.21.15.162:0/2376432314 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:18.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.518+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315522353 0 0) 0x7fa9980b28a0 con 0x7fa9980a98f0 2024-03-25T18:20:18.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.519+0000 7fa9977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:18.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.519+0000 7fa9977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:18.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.519+0000 7fa9977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:18.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.519+0000 7fa9977fe640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:20:18.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.519+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa988003520 con 0x7fa9980a98f0 2024-03-25T18:20:18.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.520+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202139431 0 0) 0x7fa9980b2a30 con 0x7fa9981430c0 2024-03-25T18:20:18.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.520+0000 7fa9977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:18.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.520+0000 7fa9977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:18.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.520+0000 7fa9977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:18.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.520+0000 7fa9977fe640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:20:18.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.520+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9980b28a0 con 0x7fa9981430c0 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887466723 0 0) 0x7fa9980b28a0 con 0x7fa9981430c0 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 >> v1:172.21.15.191:6789/0 conn(0x7fa9980a98f0 legacy=0x7fa9980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 >> v1:172.21.15.162:6789/0 conn(0x7fa99813f710 legacy=0x7fa9981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 1 monclient: found mon.noname-b 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980b44c0 con 0x7fa9981430c0 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:18.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa9900019f0 con 0x7fa9981430c0 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa9977fe640 10 monclient: dump: 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:18.530 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:18.531 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:18.531 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:18.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa99ec0f640 5 monclient: authenticate success, global_id 4499 2024-03-25T18:20:18.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.527+0000 7fa99ec0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:18.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.545+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa990001b50 con 0x7fa9981430c0 2024-03-25T18:20:18.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.545+0000 7fa9977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:18.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.545+0000 7fa9977fe640 1 -- 172.21.15.162:0/2376432314 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa990001f20 con 0x7fa9981430c0 2024-03-25T18:20:18.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.545+0000 7fa9977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:18.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.545+0000 7fa9977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.545+0000 7fa9977fe640 10 monclient: dump: 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.545+0000 7fa99ec0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:18.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.545+0000 7fa99ec0f640 4 set_mon_vals no callback set 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.546+0000 7fa99ec0f640 10 monclient: shutdown 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.546+0000 7fa99ec0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.546+0000 7fa99ec0f640 1 -- 172.21.15.162:0/2376432314 >> v1:172.21.15.162:6790/0 conn(0x7fa9981430c0 legacy=0x7fa99814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.546+0000 7fa99ec0f640 1 -- 172.21.15.162:0/2376432314 shutdown_connections 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.546+0000 7fa99ec0f640 1 -- 172.21.15.162:0/2376432314 >> 172.21.15.162:0/2376432314 conn(0x7fa9980b7ea0 msgr2=0x7fa9980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.546+0000 7fa99ec0f640 1 -- 172.21.15.162:0/2376432314 shutdown_connections 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.546+0000 7fa99ec0f640 1 -- 172.21.15.162:0/2376432314 wait complete. 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 10 monclient: build_initial_monmap 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 10 monclient: monmap: 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:18.550 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 1 librados: starting msgr at 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 1 librados: starting objecter 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 1 Processor -- start 2024-03-25T18:20:18.551 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 1 -- start start 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 1 librados: setting wanted keys 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 1 librados: calling monclient init 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.547+0000 7fa99ec0f640 10 monclient: init 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 10 monclient(hunting): picked mon.noname-a con 0x7fa9980a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 10 monclient(hunting): picked mon.noname-b con 0x7fa99813f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 10 monclient(hunting): picked mon.noname-c con 0x7fa9981430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980b02b0 con 0x7fa9980a98f0 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980b0500 con 0x7fa99813f710 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980aea50 con 0x7fa9981430c0 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:18.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.548+0000 7fa99ec0f640 10 monclient(hunting): authenticate will time out at 7917.822152s 2024-03-25T18:20:18.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.554+0000 7fa997fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa99813f710 0x7fa9980aef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:33444/0 (socket says 172.21.15.162:33444) 2024-03-25T18:20:18.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.554+0000 7fa997fff640 1 -- 172.21.15.162:0/11641763 learned_addr learned my addr 172.21.15.162:0/11641763 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:18.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.555+0000 7fa9957fa640 10 client.?.objecter ms_handle_connect 0x7fa9981430c0 2024-03-25T18:20:18.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.555+0000 7fa9957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:18.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.555+0000 7fa9957fa640 10 client.?.objecter ms_handle_connect 0x7fa99813f710 2024-03-25T18:20:18.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.555+0000 7fa9957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.563+0000 7fa9957fa640 10 client.?.objecter ms_handle_connect 0x7fa9980a98f0 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.563+0000 7fa9957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.563+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851645691 0 0) 0x7fa9980aea50 con 0x7fa9981430c0 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.563+0000 7fa9957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.563+0000 7fa9957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.563+0000 7fa9957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.563+0000 7fa9957fa640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.563+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa97c004eb0 con 0x7fa9981430c0 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.564+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970557866 0 0) 0x7fa9980b0500 con 0x7fa99813f710 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.564+0000 7fa9957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:18.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.564+0000 7fa9957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:18.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.564+0000 7fa9957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:18.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.564+0000 7fa9957fa640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:20:18.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.564+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9980aea50 con 0x7fa99813f710 2024-03-25T18:20:18.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.580+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237387378 0 0) 0x7fa9980b02b0 con 0x7fa9980a98f0 2024-03-25T18:20:18.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.580+0000 7fa9957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:18.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.580+0000 7fa9957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:18.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.580+0000 7fa9957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:18.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.580+0000 7fa9957fa640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:20:18.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.580+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9980b0500 con 0x7fa9980a98f0 2024-03-25T18:20:18.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.589+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3563353304 0 0) 0x7fa9980aea50 con 0x7fa99813f710 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.589+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9980b02b0 con 0x7fa99813f710 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.589+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa984001970 con 0x7fa99813f710 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.589+0000 7fa9957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.589+0000 7fa9957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.589+0000 7fa9957fa640 10 monclient(hunting): dump: 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:18.591 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:18.592 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:18.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.589+0000 7fa9957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:18.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.590+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3661211916 0 0) 0x7fa9980b0500 con 0x7fa9980a98f0 2024-03-25T18:20:18.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.590+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9980aea50 con 0x7fa9980a98f0 2024-03-25T18:20:18.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.590+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa98c001950 con 0x7fa9980a98f0 2024-03-25T18:20:18.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.590+0000 7fa9957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:18.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.590+0000 7fa9957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.590+0000 7fa9957fa640 10 monclient(hunting): dump: 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:18.593 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:18.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.595+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2763554091 0 0) 0x7fa97c004eb0 con 0x7fa9981430c0 2024-03-25T18:20:18.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.595+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9980b0500 con 0x7fa9981430c0 2024-03-25T18:20:18.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.595+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa990001e60 con 0x7fa9981430c0 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.595+0000 7fa9957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.595+0000 7fa9957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.595+0000 7fa9957fa640 10 monclient(hunting): dump: 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:18.597 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:18.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.606+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3482750759 0 0) 0x7fa9980b0500 con 0x7fa9981430c0 2024-03-25T18:20:18.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.606+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 >> v1:172.21.15.191:6789/0 conn(0x7fa99813f710 legacy=0x7fa9980aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 >> v1:172.21.15.162:6789/0 conn(0x7fa9980a98f0 legacy=0x7fa9980ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:18.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 1 monclient: found mon.c 2024-03-25T18:20:18.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:18.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980af680 con 0x7fa9981430c0 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa9957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 5 monclient: authenticate success, global_id 4502 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 10 monclient: _renew_subs 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa998133780 con 0x7fa9981430c0 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 10 monclient: _renew_subs 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9980ab8b0 con 0x7fa9981430c0 2024-03-25T18:20:18.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.607+0000 7fa99ec0f640 1 librados: init done 2024-03-25T18:20:18.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.608+0000 7fa99ec0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:18.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.608+0000 7fa99ec0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:18.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.608+0000 7fa99ec0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:18.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.608+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa998074cc0 con 0x7fa9981430c0 2024-03-25T18:20:18.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.935+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9900020a0 con 0x7fa9981430c0 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.935+0000 7fa9957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.935+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa990004fb0 con 0x7fa9981430c0 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.935+0000 7fa9957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.935+0000 7fa9957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.935+0000 7fa9957fa640 10 monclient: dump: 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:18.938 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:18.939 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:18.939 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:18.939 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:18.939 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:18.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.935+0000 7fa996ffd640 4 set_mon_vals no callback set 2024-03-25T18:20:18.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.936+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fa99001af80 con 0x7fa9981430c0 2024-03-25T18:20:18.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.937+0000 7fa99c9ab640 10 monclient: get_auth_request con 0x7fa97c06e0c0 auth_method 0 2024-03-25T18:20:18.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.937+0000 7fa9957fa640 10 client.4502.objecter ms_handle_connect 0x7fa97c06e0c0 2024-03-25T18:20:18.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.942+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fa990054bf0 con 0x7fa9981430c0 2024-03-25T18:20:18.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.942+0000 7fa9957fa640 10 client.4502.objecter ms_dispatch 0x7fa998000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:18.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.942+0000 7fa9957fa640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:18.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.942+0000 7fa9957fa640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:18.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.942+0000 7fa9957fa640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-25T18:20:18.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.942+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa990050530 con 0x7fa9981430c0 2024-03-25T18:20:18.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.942+0000 7fa9957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:18.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:18.942+0000 7fa9957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:19.166 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.165+0000 7fa99ec0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:19.166 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.165+0000 7fa99ec0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:19.166 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.165+0000 7fa99ec0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:19.166 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.165+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9980aebe0 con 0x7fa9981430c0 2024-03-25T18:20:19.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.272+0000 7fa9957fa640 1 -- 172.21.15.162:0/11641763 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fa990052630 con 0x7fa9981430c0 2024-03-25T18:20:19.274 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.272+0000 7fa9957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:19.274 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.272+0000 7fa9957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:19.274 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:20:19.274 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 10 librados: watch_flush enter 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 10 librados: watch_flush exit 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 10 client.4502.objecter successfully canceled tick 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 >> v1:172.21.15.191:6800/1552703795 conn(0x7fa97c06e0c0 legacy=0x7fa97c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 10 monclient: shutdown 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 >> v1:172.21.15.162:6790/0 conn(0x7fa9981430c0 legacy=0x7fa998133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 shutdown_connections 2024-03-25T18:20:19.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 >> 172.21.15.162:0/11641763 conn(0x7fa9980b7ea0 msgr2=0x7fa998074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:19.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 shutdown_connections 2024-03-25T18:20:19.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.275+0000 7fa99ec0f640 1 -- 172.21.15.162:0/11641763 wait complete. 2024-03-25T18:20:19.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:19.276+0000 7fa99ec0f640 1 librados: shutdown 2024-03-25T18:20:19.288 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:26.094 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:26.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.205+0000 7fa3be065640 10 monclient: get_monmap_and_config 2024-03-25T18:20:26.206 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.205+0000 7fa3be065640 10 monclient: build_initial_monmap 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.205+0000 7fa3be065640 10 monclient: monmap: 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:20:26.206594+0000 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:20:26.206594+0000 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:20:26.207 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:26.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.206+0000 7fa3be065640 1 Processor -- start 2024-03-25T18:20:26.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.206+0000 7fa3be065640 1 -- start start 2024-03-25T18:20:26.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.206+0000 7fa3be065640 10 monclient: init 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.206+0000 7fa3be065640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.206+0000 7fa3be065640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.207+0000 7fa3be065640 10 monclient(hunting): picked mon.noname-c con 0x7fa3b80a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.207+0000 7fa3be065640 10 monclient(hunting): picked mon.noname-b con 0x7fa3b813f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.207+0000 7fa3be065640 10 monclient(hunting): picked mon.noname-a con 0x7fa3b81430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.207+0000 7fa3be065640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b8142520 con 0x7fa3b81430c0 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.207+0000 7fa3be065640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b80b28a0 con 0x7fa3b80a98f0 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.207+0000 7fa3be065640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b80b2a30 con 0x7fa3b813f710 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.207+0000 7fa3be065640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:26.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.207+0000 7fa3be065640 10 monclient(hunting): authenticate will time out at 7925.480826s 2024-03-25T18:20:26.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.210+0000 7fa3b6ffd640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fa3b813f710 0x7fa3b81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:55632/0 (socket says 172.21.15.162:55632) 2024-03-25T18:20:26.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.210+0000 7fa3b6ffd640 1 -- 172.21.15.162:0/853560242 learned_addr learned my addr 172.21.15.162:0/853560242 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:26.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.224+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911538504 0 0) 0x7fa3b80b2a30 con 0x7fa3b813f710 2024-03-25T18:20:26.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.224+0000 7fa3b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.224+0000 7fa3b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.224+0000 7fa3b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.224+0000 7fa3b67fc640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:20:26.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.224+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a0003520 con 0x7fa3b813f710 2024-03-25T18:20:26.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.228+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238790923 0 0) 0x7fa3b80b28a0 con 0x7fa3b80a98f0 2024-03-25T18:20:26.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.228+0000 7fa3b67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.228+0000 7fa3b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.228+0000 7fa3b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.228+0000 7fa3b67fc640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:20:26.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.228+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b80b2a30 con 0x7fa3b80a98f0 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 464326584 0 0) 0x7fa3a0003520 con 0x7fa3b813f710 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 >> v1:172.21.15.191:6789/0 conn(0x7fa3b80a98f0 legacy=0x7fa3b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 >> v1:172.21.15.162:6789/0 conn(0x7fa3b81430c0 legacy=0x7fa3b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 1 monclient: found mon.noname-b 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b80b44c0 con 0x7fa3b813f710 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:26.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa3ac001690 con 0x7fa3b813f710 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3be065640 5 monclient: authenticate success, global_id 4505 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3be065640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3b67fc640 10 monclient: dump: 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:26.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.241+0000 7fa3be065640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:26.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.274+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3ac0017f0 con 0x7fa3b813f710 2024-03-25T18:20:26.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.274+0000 7fa3b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:26.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.274+0000 7fa3b67fc640 1 -- 172.21.15.162:0/853560242 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa3ac001c00 con 0x7fa3b813f710 2024-03-25T18:20:26.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.274+0000 7fa3be065640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:26.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.274+0000 7fa3be065640 4 set_mon_vals no callback set 2024-03-25T18:20:26.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.274+0000 7fa3b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.274+0000 7fa3b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:26.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.274+0000 7fa3b67fc640 10 monclient: dump: 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:26.276 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:26.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.275+0000 7fa3be065640 10 monclient: shutdown 2024-03-25T18:20:26.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.275+0000 7fa3be065640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:26.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.275+0000 7fa3be065640 1 -- 172.21.15.162:0/853560242 >> v1:172.21.15.162:6790/0 conn(0x7fa3b813f710 legacy=0x7fa3b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.275+0000 7fa3be065640 1 -- 172.21.15.162:0/853560242 shutdown_connections 2024-03-25T18:20:26.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.275+0000 7fa3be065640 1 -- 172.21.15.162:0/853560242 >> 172.21.15.162:0/853560242 conn(0x7fa3b80b7ea0 msgr2=0x7fa3b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:26.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.275+0000 7fa3be065640 1 -- 172.21.15.162:0/853560242 shutdown_connections 2024-03-25T18:20:26.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.275+0000 7fa3be065640 1 -- 172.21.15.162:0/853560242 wait complete. 2024-03-25T18:20:26.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.276+0000 7fa3be065640 10 monclient: build_initial_monmap 2024-03-25T18:20:26.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.276+0000 7fa3be065640 10 monclient: monmap: 2024-03-25T18:20:26.278 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:26.278 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:26.278 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:20:26.278 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:20:26.278 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:26.278 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.277+0000 7fa3be065640 1 librados: starting msgr at 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.277+0000 7fa3be065640 1 librados: starting objecter 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.277+0000 7fa3be065640 1 Processor -- start 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.277+0000 7fa3be065640 1 -- start start 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.277+0000 7fa3be065640 1 librados: setting wanted keys 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.277+0000 7fa3be065640 1 librados: calling monclient init 2024-03-25T18:20:26.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.277+0000 7fa3be065640 10 monclient: init 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 10 monclient(hunting): picked mon.noname-a con 0x7fa3b80a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 10 monclient(hunting): picked mon.noname-b con 0x7fa3b813f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 10 monclient(hunting): picked mon.noname-c con 0x7fa3b81430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b813be70 con 0x7fa3b80a98f0 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b8131660 con 0x7fa3b813f710 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b81317f0 con 0x7fa3b81430c0 2024-03-25T18:20:26.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:26.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.278+0000 7fa3be065640 10 monclient(hunting): authenticate will time out at 7925.552203s 2024-03-25T18:20:26.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.291+0000 7fa3b7fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fa3b81430c0 0x7fa3b81e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:55692/0 (socket says 172.21.15.162:55692) 2024-03-25T18:20:26.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.291+0000 7fa3b7fff640 1 -- 172.21.15.162:0/2882372475 learned_addr learned my addr 172.21.15.162:0/2882372475 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:26.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.293+0000 7fa39ffff640 10 client.?.objecter ms_handle_connect 0x7fa3b81430c0 2024-03-25T18:20:26.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.293+0000 7fa39ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:26.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.298+0000 7fa39ffff640 10 client.?.objecter ms_handle_connect 0x7fa3b80a98f0 2024-03-25T18:20:26.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.298+0000 7fa39ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:26.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.299+0000 7fa39ffff640 10 client.?.objecter ms_handle_connect 0x7fa3b813f710 2024-03-25T18:20:26.300 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.299+0000 7fa39ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:26.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886832997 0 0) 0x7fa3b81317f0 con 0x7fa3b81430c0 2024-03-25T18:20:26.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:20:26.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a4004eb0 con 0x7fa3b81430c0 2024-03-25T18:20:26.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500147043 0 0) 0x7fa3b8131660 con 0x7fa3b813f710 2024-03-25T18:20:26.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:20:26.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.312+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b81317f0 con 0x7fa3b813f710 2024-03-25T18:20:26.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.317+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328997194 0 0) 0x7fa3b813be70 con 0x7fa3b80a98f0 2024-03-25T18:20:26.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.317+0000 7fa39ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:26.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.317+0000 7fa39ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:26.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.317+0000 7fa39ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:26.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.317+0000 7fa39ffff640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:20:26.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.317+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3b8131660 con 0x7fa3b80a98f0 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.334+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3003538735 0 0) 0x7fa3b8131660 con 0x7fa3b80a98f0 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.334+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3b813be70 con 0x7fa3b80a98f0 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.334+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa3a8001930 con 0x7fa3b80a98f0 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.334+0000 7fa39ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.334+0000 7fa39ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.334+0000 7fa39ffff640 10 monclient(hunting): dump: 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.335 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:26.336 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:26.336 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:26.336 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:26.336 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:26.336 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.334+0000 7fa39ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.341+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2014070951 0 0) 0x7fa3a4004eb0 con 0x7fa3b81430c0 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.341+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3b8131660 con 0x7fa3b81430c0 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.341+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa3b0001c90 con 0x7fa3b81430c0 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.341+0000 7fa39ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.341+0000 7fa39ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.341+0000 7fa39ffff640 10 monclient(hunting): dump: 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:26.343 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:26.344 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:26.352 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2286235555 0 0) 0x7fa3b813be70 con 0x7fa3b80a98f0 2024-03-25T18:20:26.352 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 >> v1:172.21.15.162:6790/0 conn(0x7fa3b81430c0 legacy=0x7fa3b81e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.352 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 >> v1:172.21.15.191:6789/0 conn(0x7fa3b813f710 legacy=0x7fa3b81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 1 monclient: found mon.a 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8131980 con 0x7fa3b80a98f0 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 10 monclient: _finish_auth 0 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa39ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa3be065640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa3be065640 10 monclient: _renew_subs 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa3be065640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b8135c60 con 0x7fa3b80a98f0 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa3be065640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa3be065640 10 monclient: _renew_subs 2024-03-25T18:20:26.353 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa3be065640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:26.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.350+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b80b7770 con 0x7fa3b80a98f0 2024-03-25T18:20:26.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.351+0000 7fa3be065640 1 librados: init done 2024-03-25T18:20:26.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.352+0000 7fa3be065640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:26.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.352+0000 7fa3be065640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:26.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.352+0000 7fa3be065640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:26.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.352+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b81371d0 con 0x7fa3b80a98f0 2024-03-25T18:20:26.665 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.664+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3a80035f0 con 0x7fa3b80a98f0 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.664+0000 7fa39ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.664+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa3a8001d80 con 0x7fa3b80a98f0 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.664+0000 7fa39ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.664+0000 7fa39ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.664+0000 7fa39ffff640 10 monclient: dump: 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:26.666 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:26.667 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:26.667 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:26.667 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:26.667 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:26.667 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:26.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.664+0000 7fa3b57fa640 4 set_mon_vals no callback set 2024-03-25T18:20:26.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.665+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fa3a8019830 con 0x7fa3b80a98f0 2024-03-25T18:20:26.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.665+0000 7fa3b6ffd640 10 monclient: get_auth_request con 0x7fa3a406dfb0 auth_method 0 2024-03-25T18:20:26.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.665+0000 7fa39ffff640 10 client.4521.objecter ms_handle_connect 0x7fa3a406dfb0 2024-03-25T18:20:26.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.671+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fa3a8053560 con 0x7fa3b80a98f0 2024-03-25T18:20:26.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.671+0000 7fa39ffff640 10 client.4521.objecter ms_dispatch 0x7fa3b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:26.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.671+0000 7fa39ffff640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:26.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.671+0000 7fa39ffff640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:26.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.671+0000 7fa39ffff640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T18:20:26.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.671+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3a804eea0 con 0x7fa3b80a98f0 2024-03-25T18:20:26.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.671+0000 7fa39ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:26.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.671+0000 7fa39ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:26.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.907+0000 7fa3be065640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.907+0000 7fa3be065640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.907+0000 7fa3be065640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:26.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.907+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3b8074cc0 con 0x7fa3b80a98f0 2024-03-25T18:20:26.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.924+0000 7fa39ffff640 1 -- 172.21.15.162:0/2882372475 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fa3a8050fa0 con 0x7fa3b80a98f0 2024-03-25T18:20:26.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.924+0000 7fa39ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.924+0000 7fa39ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:26.926 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:20:26.926 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:26.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 10 librados: watch_flush enter 2024-03-25T18:20:26.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 10 librados: watch_flush exit 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 10 client.4521.objecter successfully canceled tick 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 >> v1:172.21.15.191:6800/1552703795 conn(0x7fa3a406dfb0 legacy=0x7fa3a4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 10 monclient: shutdown 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 >> v1:172.21.15.162:6789/0 conn(0x7fa3b80a98f0 legacy=0x7fa3b8130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 shutdown_connections 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 >> 172.21.15.162:0/2882372475 conn(0x7fa3b80b7ea0 msgr2=0x7fa3b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 shutdown_connections 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 1 -- 172.21.15.162:0/2882372475 wait complete. 2024-03-25T18:20:26.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:26.927+0000 7fa3be065640 1 librados: shutdown 2024-03-25T18:20:26.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:33.746 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.862+0000 7f64a10ba640 10 monclient: get_monmap_and_config 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.862+0000 7f64a10ba640 10 monclient: build_initial_monmap 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.862+0000 7f64a10ba640 10 monclient: monmap: 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:20:33.863867+0000 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:20:33.863867+0000 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:33.864 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:33.865 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:33.865 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:20:33.865 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:20:33.865 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:33.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.863+0000 7f64a10ba640 1 Processor -- start 2024-03-25T18:20:33.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.863+0000 7f64a10ba640 1 -- start start 2024-03-25T18:20:33.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.863+0000 7f64a10ba640 10 monclient: init 2024-03-25T18:20:33.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:33.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:33.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 10 monclient(hunting): picked mon.noname-b con 0x7f649c0a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:33.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 10 monclient(hunting): picked mon.noname-a con 0x7f649c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:33.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 10 monclient(hunting): picked mon.noname-c con 0x7f649c1430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:33.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c142520 con 0x7f649c13f710 2024-03-25T18:20:33.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0b28a0 con 0x7f649c1430c0 2024-03-25T18:20:33.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0b2a30 con 0x7f649c0a98f0 2024-03-25T18:20:33.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:33.867 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.864+0000 7f64a10ba640 10 monclient(hunting): authenticate will time out at 7933.138202s 2024-03-25T18:20:33.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.871+0000 7f649ad9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f649c0a98f0 0x7f649c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:57696/0 (socket says 172.21.15.162:57696) 2024-03-25T18:20:33.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.871+0000 7f649ad9d640 1 -- 172.21.15.162:0/3656789790 learned_addr learned my addr 172.21.15.162:0/3656789790 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:33.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.893+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580811506 0 0) 0x7f649c0b2a30 con 0x7f649c0a98f0 2024-03-25T18:20:33.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.893+0000 7f649a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.893+0000 7f649a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.893+0000 7f649a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.893+0000 7f649a59c640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:20:33.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.893+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6488003520 con 0x7f649c0a98f0 2024-03-25T18:20:33.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.898+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598598257 0 0) 0x7f649c142520 con 0x7f649c13f710 2024-03-25T18:20:33.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.898+0000 7f649a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.898+0000 7f649a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.898+0000 7f649a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.898+0000 7f649a59c640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:20:33.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.898+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f649c0b2a30 con 0x7f649c13f710 2024-03-25T18:20:33.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.899+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156437093 0 0) 0x7f649c0b28a0 con 0x7f649c1430c0 2024-03-25T18:20:33.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.899+0000 7f649a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.899+0000 7f649a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.899+0000 7f649a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.899+0000 7f649a59c640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:20:33.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.899+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f649c142520 con 0x7f649c1430c0 2024-03-25T18:20:33.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749529156 0 0) 0x7f6488003520 con 0x7f649c0a98f0 2024-03-25T18:20:33.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 >> v1:172.21.15.191:6789/0 conn(0x7f649c1430c0 legacy=0x7f649c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 >> v1:172.21.15.162:6789/0 conn(0x7f649c13f710 legacy=0x7f649c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 10 monclient: _finish_hunting 0 2024-03-25T18:20:33.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 1 monclient: found mon.noname-b 2024-03-25T18:20:33.914 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0b44c0 con 0x7f649c0a98f0 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 10 monclient: _finish_auth 0 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 10 monclient: _check_auth_tickets 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f648c001690 con 0x7f649c0a98f0 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f649a59c640 10 monclient: dump: 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:33.915 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:33.916 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:33.916 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:33.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f64a10ba640 5 monclient: authenticate success, global_id 4511 2024-03-25T18:20:33.916 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.913+0000 7f64a10ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.922+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f648c0017f0 con 0x7f649c0a98f0 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.922+0000 7f649a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f649a59c640 1 -- 172.21.15.162:0/3656789790 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f648c001c00 con 0x7f649c0a98f0 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f64a10ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f64a10ba640 4 set_mon_vals no callback set 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f649a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f649a59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f649a59c640 10 monclient: dump: 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:33.924 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:33.925 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.925 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:33.925 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:33.925 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:33.925 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:33.925 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:33.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f64a10ba640 10 monclient: shutdown 2024-03-25T18:20:33.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f64a10ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:33.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f64a10ba640 1 -- 172.21.15.162:0/3656789790 >> v1:172.21.15.162:6790/0 conn(0x7f649c0a98f0 legacy=0x7f649c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f64a10ba640 1 -- 172.21.15.162:0/3656789790 shutdown_connections 2024-03-25T18:20:33.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.923+0000 7f64a10ba640 1 -- 172.21.15.162:0/3656789790 >> 172.21.15.162:0/3656789790 conn(0x7f649c0b7ea0 msgr2=0x7f649c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:33.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.924+0000 7f64a10ba640 1 -- 172.21.15.162:0/3656789790 shutdown_connections 2024-03-25T18:20:33.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.924+0000 7f64a10ba640 1 -- 172.21.15.162:0/3656789790 wait complete. 2024-03-25T18:20:33.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.925+0000 7f64a10ba640 10 monclient: build_initial_monmap 2024-03-25T18:20:33.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.925+0000 7f64a10ba640 10 monclient: monmap: 2024-03-25T18:20:33.927 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:33.927 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:33.927 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:20:33.927 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:20:33.927 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:33.927 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.925+0000 7f64a10ba640 1 librados: starting msgr at 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.925+0000 7f64a10ba640 1 librados: starting objecter 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 1 Processor -- start 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 1 -- start start 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 1 librados: setting wanted keys 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 1 librados: calling monclient init 2024-03-25T18:20:33.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 10 monclient: init 2024-03-25T18:20:33.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:33.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:33.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 10 monclient(hunting): picked mon.noname-a con 0x7f649c0a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:33.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 10 monclient(hunting): picked mon.noname-b con 0x7f649c13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:33.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.926+0000 7f64a10ba640 10 monclient(hunting): picked mon.noname-c con 0x7f649c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:33.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.927+0000 7f64a10ba640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0ab9e0 con 0x7f649c0a98f0 2024-03-25T18:20:33.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.927+0000 7f64a10ba640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c138c20 con 0x7f649c13f710 2024-03-25T18:20:33.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.927+0000 7f64a10ba640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c138f00 con 0x7f649c1430c0 2024-03-25T18:20:33.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.927+0000 7f64a10ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:33.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.927+0000 7f64a10ba640 10 monclient(hunting): authenticate will time out at 7933.200793s 2024-03-25T18:20:33.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.933+0000 7f649b59e640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f649c1430c0 0x7f649c1e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:57730/0 (socket says 172.21.15.162:57730) 2024-03-25T18:20:33.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.933+0000 7f649b59e640 1 -- 172.21.15.162:0/2492797636 learned_addr learned my addr 172.21.15.162:0/2492797636 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:33.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.937+0000 7f64937fe640 10 client.?.objecter ms_handle_connect 0x7f649c0a98f0 2024-03-25T18:20:33.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.938+0000 7f64937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.938+0000 7f64937fe640 10 client.?.objecter ms_handle_connect 0x7f649c1430c0 2024-03-25T18:20:33.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.938+0000 7f64937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.945+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282266774 0 0) 0x7f649c0ab9e0 con 0x7f649c0a98f0 2024-03-25T18:20:33.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.945+0000 7f64937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.945+0000 7f64937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.945+0000 7f64937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.945+0000 7f64937fe640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:20:33.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.945+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c004eb0 con 0x7f649c0a98f0 2024-03-25T18:20:33.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.950+0000 7f64937fe640 10 client.?.objecter ms_handle_connect 0x7f649c13f710 2024-03-25T18:20:33.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.950+0000 7f64937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.953+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2603614481 0 0) 0x7f647c004eb0 con 0x7f649c0a98f0 2024-03-25T18:20:33.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.953+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f649c0ab9e0 con 0x7f649c0a98f0 2024-03-25T18:20:33.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.953+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f648c001b00 con 0x7f649c0a98f0 2024-03-25T18:20:33.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.953+0000 7f64937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.953+0000 7f64937fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:33.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.953+0000 7f64937fe640 10 monclient(hunting): dump: 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:33.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.953+0000 7f64937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:33.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.956+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211757176 0 0) 0x7f649c138f00 con 0x7f649c1430c0 2024-03-25T18:20:33.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.956+0000 7f64937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.956+0000 7f64937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.956+0000 7f64937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.956+0000 7f64937fe640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:20:33.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.956+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c004eb0 con 0x7f649c1430c0 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3585517845 0 0) 0x7f649c0ab9e0 con 0x7f649c0a98f0 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 >> v1:172.21.15.162:6790/0 conn(0x7f649c1430c0 legacy=0x7f649c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 >> v1:172.21.15.191:6789/0 conn(0x7f649c13f710 legacy=0x7f649c1d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 1 monclient: found mon.a 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c14e7a0 con 0x7f649c0a98f0 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:33.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:20:33.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 10 monclient: _renew_subs 2024-03-25T18:20:33.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:33.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c1386d0 con 0x7f649c0a98f0 2024-03-25T18:20:33.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:33.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 10 monclient: _renew_subs 2024-03-25T18:20:33.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:33.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c1d8060 con 0x7f649c0a98f0 2024-03-25T18:20:33.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.962+0000 7f64a10ba640 1 librados: init done 2024-03-25T18:20:33.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.963+0000 7f64a10ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.963+0000 7f64a10ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.963+0000 7f64a10ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:33.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:33.963+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c0627b0 con 0x7f649c0a98f0 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.274+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f648c001220 con 0x7f649c0a98f0 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.274+0000 7f64937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.274+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f648c004c30 con 0x7f649c0a98f0 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.274+0000 7f64937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.274+0000 7f64937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.274+0000 7f64937fe640 10 monclient: dump: 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:34.276 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:34.277 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:34.277 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:34.277 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:34.277 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:34.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.274+0000 7f6499d9b640 4 set_mon_vals no callback set 2024-03-25T18:20:34.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.275+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f648c01ac40 con 0x7f649c0a98f0 2024-03-25T18:20:34.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.276+0000 7f6493fff640 10 monclient: get_auth_request con 0x7f647c06dd40 auth_method 0 2024-03-25T18:20:34.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.276+0000 7f64937fe640 10 client.4527.objecter ms_handle_connect 0x7f647c06dd40 2024-03-25T18:20:34.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.281+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f648c0548a0 con 0x7f649c0a98f0 2024-03-25T18:20:34.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.282+0000 7f64937fe640 10 client.4527.objecter ms_dispatch 0x7f649c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:34.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.282+0000 7f64937fe640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:34.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.282+0000 7f64937fe640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:34.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.282+0000 7f64937fe640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T18:20:34.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.282+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f648c0501e0 con 0x7f649c0a98f0 2024-03-25T18:20:34.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.282+0000 7f64937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:34.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.282+0000 7f64937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:34.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.541+0000 7f64a10ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:34.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.542+0000 7f64a10ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:34.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.542+0000 7f64a10ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:20:34.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.542+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f649c069f40 con 0x7f649c0a98f0 2024-03-25T18:20:34.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.578+0000 7f64937fe640 1 -- 172.21.15.162:0/2492797636 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f648c0522e0 con 0x7f649c0a98f0 2024-03-25T18:20:34.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.578+0000 7f64937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:34.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.578+0000 7f64937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:34.580 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:20:34.580 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 10 librados: watch_flush enter 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 10 librados: watch_flush exit 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 10 client.4527.objecter successfully canceled tick 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 >> v1:172.21.15.191:6800/1552703795 conn(0x7f647c06dd40 legacy=0x7f647c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 10 monclient: shutdown 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 >> v1:172.21.15.162:6789/0 conn(0x7f649c0a98f0 legacy=0x7f649c137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:34.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 shutdown_connections 2024-03-25T18:20:34.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 >> 172.21.15.162:0/2492797636 conn(0x7f649c0b7ea0 msgr2=0x7f649c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:34.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 shutdown_connections 2024-03-25T18:20:34.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 1 -- 172.21.15.162:0/2492797636 wait complete. 2024-03-25T18:20:34.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:34.581+0000 7f64a10ba640 1 librados: shutdown 2024-03-25T18:20:34.595 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:38.062 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:50.289 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:38.077 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:50.305 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:38.125 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:06:50.353 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:38.128 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:50.355 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:38.133 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:50.349 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:38.134 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:50.361 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:38.145 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:50.361 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:38.173 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:06:50.400 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:38.177 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:06:50.393 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:41.402 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:41.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.520+0000 7febe44c8640 10 monclient: get_monmap_and_config 2024-03-25T18:20:41.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.520+0000 7febe44c8640 10 monclient: build_initial_monmap 2024-03-25T18:20:41.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.520+0000 7febe44c8640 10 monclient: monmap: 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:20:41.521406+0000 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:20:41.521406+0000 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:20:41.522 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:41.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.521+0000 7febe44c8640 1 Processor -- start 2024-03-25T18:20:41.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.521+0000 7febe44c8640 1 -- start start 2024-03-25T18:20:41.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.521+0000 7febe44c8640 10 monclient: init 2024-03-25T18:20:41.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 10 monclient(hunting): picked mon.noname-b con 0x7febdc0a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 10 monclient(hunting): picked mon.noname-c con 0x7febdc13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 10 monclient(hunting): picked mon.noname-a con 0x7febdc1430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc142520 con 0x7febdc1430c0 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0b28a0 con 0x7febdc13f710 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc0b2a30 con 0x7febdc0a98f0 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:41.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.522+0000 7febe44c8640 10 monclient(hunting): authenticate will time out at 7940.795859s 2024-03-25T18:20:41.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.537+0000 7febe2a65640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7febdc1430c0 0x7febdc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:42396/0 (socket says 172.21.15.162:42396) 2024-03-25T18:20:41.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.537+0000 7febe2a65640 1 -- 172.21.15.162:0/1996585210 learned_addr learned my addr 172.21.15.162:0/1996585210 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:41.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.547+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636706344 0 0) 0x7febdc142520 con 0x7febdc1430c0 2024-03-25T18:20:41.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.547+0000 7febe1262640 10 monclient(hunting): _negotiate 2024-03-25T18:20:41.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.547+0000 7febe1262640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:41.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.547+0000 7febe1262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:41.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.547+0000 7febe1262640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:20:41.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.547+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc8003520 con 0x7febdc1430c0 2024-03-25T18:20:41.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.556+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194007689 0 0) 0x7febdc0b2a30 con 0x7febdc0a98f0 2024-03-25T18:20:41.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.556+0000 7febe1262640 10 monclient(hunting): _negotiate 2024-03-25T18:20:41.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.556+0000 7febe1262640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:41.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.556+0000 7febe1262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:41.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.556+0000 7febe1262640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:20:41.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.556+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc142520 con 0x7febdc0a98f0 2024-03-25T18:20:41.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1075852780 0 0) 0x7febc8003520 con 0x7febdc1430c0 2024-03-25T18:20:41.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 >> v1:172.21.15.162:6790/0 conn(0x7febdc0a98f0 legacy=0x7febdc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 >> v1:172.21.15.191:6789/0 conn(0x7febdc13f710 legacy=0x7febdc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 10 monclient: _finish_hunting 0 2024-03-25T18:20:41.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 1 monclient: found mon.noname-a 2024-03-25T18:20:41.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:41.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc0b44c0 con 0x7febdc1430c0 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 10 monclient: _finish_auth 0 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 10 monclient: _check_auth_tickets 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7febd4001a10 con 0x7febdc1430c0 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe1262640 10 monclient: dump: 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:41.566 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:41.567 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:41.567 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:41.567 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:41.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe44c8640 5 monclient: authenticate success, global_id 4530 2024-03-25T18:20:41.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.564+0000 7febe44c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:41.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.587+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febd4001b70 con 0x7febdc1430c0 2024-03-25T18:20:41.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.587+0000 7febe1262640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:41.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.587+0000 7febe1262640 1 -- 172.21.15.162:0/1996585210 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7febd4001f80 con 0x7febdc1430c0 2024-03-25T18:20:41.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.587+0000 7febe1262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:41.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.587+0000 7febe1262640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:41.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.587+0000 7febe1262640 10 monclient: dump: 2024-03-25T18:20:41.588 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:41.588 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.587+0000 7febe44c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:41.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.587+0000 7febe44c8640 4 set_mon_vals no callback set 2024-03-25T18:20:41.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.588+0000 7febe44c8640 10 monclient: shutdown 2024-03-25T18:20:41.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.588+0000 7febe44c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:41.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.588+0000 7febe44c8640 1 -- 172.21.15.162:0/1996585210 >> v1:172.21.15.162:6789/0 conn(0x7febdc1430c0 legacy=0x7febdc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.588+0000 7febe44c8640 1 -- 172.21.15.162:0/1996585210 shutdown_connections 2024-03-25T18:20:41.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.588+0000 7febe44c8640 1 -- 172.21.15.162:0/1996585210 >> 172.21.15.162:0/1996585210 conn(0x7febdc0b7ea0 msgr2=0x7febdc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:41.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.588+0000 7febe44c8640 1 -- 172.21.15.162:0/1996585210 shutdown_connections 2024-03-25T18:20:41.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.588+0000 7febe44c8640 1 -- 172.21.15.162:0/1996585210 wait complete. 2024-03-25T18:20:41.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.589+0000 7febe44c8640 10 monclient: build_initial_monmap 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.589+0000 7febe44c8640 10 monclient: monmap: 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:20:41.591 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:41.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.589+0000 7febe44c8640 1 librados: starting msgr at 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.589+0000 7febe44c8640 1 librados: starting objecter 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 1 Processor -- start 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 1 -- start start 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 1 librados: setting wanted keys 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 1 librados: calling monclient init 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 10 monclient: init 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 10 monclient(hunting): picked mon.noname-c con 0x7febdc0a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.590+0000 7febe44c8640 10 monclient(hunting): picked mon.noname-b con 0x7febdc13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.591+0000 7febe44c8640 10 monclient(hunting): picked mon.noname-a con 0x7febdc1430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.591+0000 7febe44c8640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc071f20 con 0x7febdc1430c0 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.591+0000 7febe44c8640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc14d300 con 0x7febdc13f710 2024-03-25T18:20:41.593 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.591+0000 7febe44c8640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febdc14d5e0 con 0x7febdc0a98f0 2024-03-25T18:20:41.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.591+0000 7febe44c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:41.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.591+0000 7febe44c8640 10 monclient(hunting): authenticate will time out at 7940.864796s 2024-03-25T18:20:41.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.607+0000 7febe1a63640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7febdc13f710 0x7febdc1e32b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:35616/0 (socket says 172.21.15.162:35616) 2024-03-25T18:20:41.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.607+0000 7febe1a63640 1 -- 172.21.15.162:0/1825103315 learned_addr learned my addr 172.21.15.162:0/1825103315 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:41.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.608+0000 7febd2ffd640 10 client.?.objecter ms_handle_connect 0x7febdc13f710 2024-03-25T18:20:41.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.608+0000 7febd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:41.612 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.611+0000 7febd2ffd640 10 client.?.objecter ms_handle_connect 0x7febdc0a98f0 2024-03-25T18:20:41.612 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.611+0000 7febd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:41.623 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.622+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316390433 0 0) 0x7febdc14d5e0 con 0x7febdc0a98f0 2024-03-25T18:20:41.623 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.622+0000 7febd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:41.623 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.622+0000 7febd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:41.623 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.622+0000 7febd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:41.623 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.622+0000 7febd2ffd640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:20:41.623 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.622+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc0004eb0 con 0x7febdc0a98f0 2024-03-25T18:20:41.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.624+0000 7febd2ffd640 10 client.?.objecter ms_handle_connect 0x7febdc1430c0 2024-03-25T18:20:41.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.624+0000 7febd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:41.637 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.636+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978151442 0 0) 0x7febdc071f20 con 0x7febdc1430c0 2024-03-25T18:20:41.637 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.636+0000 7febd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:41.637 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.636+0000 7febd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:41.637 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.636+0000 7febd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:41.637 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.636+0000 7febd2ffd640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:20:41.637 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.636+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc14d5e0 con 0x7febdc1430c0 2024-03-25T18:20:41.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.700+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650330858 0 0) 0x7febdc14d300 con 0x7febdc13f710 2024-03-25T18:20:41.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.700+0000 7febd2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:41.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.700+0000 7febd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:41.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.700+0000 7febd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:41.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.700+0000 7febd2ffd640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:20:41.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.700+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc071f20 con 0x7febdc13f710 2024-03-25T18:20:41.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3482235507 0 0) 0x7febdc071f20 con 0x7febdc13f710 2024-03-25T18:20:41.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febdc14d300 con 0x7febdc13f710 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7febcc0018f0 con 0x7febdc13f710 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:41.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764980769 0 0) 0x7febdc14d5e0 con 0x7febdc1430c0 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febdc071f20 con 0x7febdc1430c0 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7febd4001e80 con 0x7febdc1430c0 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.709+0000 7febd2ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:41.712 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.715+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4121335934 0 0) 0x7febc0004eb0 con 0x7febdc0a98f0 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.715+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febdc14d5e0 con 0x7febdc0a98f0 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.715+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7febd8001930 con 0x7febdc0a98f0 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.715+0000 7febd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.715+0000 7febd2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.715+0000 7febd2ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:41.716 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.717 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:41.717 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:41.717 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:41.717 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:41.717 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:41.717 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:41.717 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:41.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.724+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 299888348 0 0) 0x7febdc14d5e0 con 0x7febdc0a98f0 2024-03-25T18:20:41.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.724+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 >> v1:172.21.15.191:6789/0 conn(0x7febdc13f710 legacy=0x7febdc1e32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.724+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 >> v1:172.21.15.162:6789/0 conn(0x7febdc1430c0 legacy=0x7febdc1e69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.724+0000 7febd2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.724+0000 7febd2ffd640 1 monclient: found mon.c 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.724+0000 7febd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.724+0000 7febd2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.724+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc14d8c0 con 0x7febdc0a98f0 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febd2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febd2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 5 monclient: authenticate success, global_id 4520 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 10 monclient: _renew_subs 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febdc14dba0 con 0x7febdc0a98f0 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 10 monclient: _renew_subs 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:41.727 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febdc06dea0 con 0x7febdc0a98f0 2024-03-25T18:20:41.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.725+0000 7febe44c8640 1 librados: init done 2024-03-25T18:20:41.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.726+0000 7febe44c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:41.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.726+0000 7febe44c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:41.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.726+0000 7febe44c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:41.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:41.726+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febdc0720f0 con 0x7febdc0a98f0 2024-03-25T18:20:42.032 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.031+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febd8003610 con 0x7febdc0a98f0 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.031+0000 7febd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.031+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7febd8003920 con 0x7febdc0a98f0 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.031+0000 7febd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.031+0000 7febd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.031+0000 7febd2ffd640 10 monclient: dump: 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:42.033 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:42.034 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:42.034 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:42.034 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:42.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.031+0000 7febd3fff640 4 set_mon_vals no callback set 2024-03-25T18:20:42.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.032+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7febd8019970 con 0x7febdc0a98f0 2024-03-25T18:20:42.034 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.032+0000 7febe1a63640 10 monclient: get_auth_request con 0x7febc006e160 auth_method 0 2024-03-25T18:20:42.035 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.033+0000 7febd2ffd640 10 client.4520.objecter ms_handle_connect 0x7febc006e160 2024-03-25T18:20:42.039 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.038+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7febd80535c0 con 0x7febdc0a98f0 2024-03-25T18:20:42.039 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.038+0000 7febd2ffd640 10 client.4520.objecter ms_dispatch 0x7febdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:42.039 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.038+0000 7febd2ffd640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:42.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.038+0000 7febd2ffd640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:42.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.038+0000 7febd2ffd640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-25T18:20:42.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.038+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febd804ef00 con 0x7febdc0a98f0 2024-03-25T18:20:42.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.038+0000 7febd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:42.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.038+0000 7febd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:42.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.284+0000 7febe44c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:42.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.284+0000 7febe44c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:42.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.284+0000 7febe44c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:42.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.284+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febdc0627b0 con 0x7febdc0a98f0 2024-03-25T18:20:42.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.312+0000 7febd2ffd640 1 -- 172.21.15.162:0/1825103315 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7febd8051000 con 0x7febdc0a98f0 2024-03-25T18:20:42.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.312+0000 7febd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:42.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.312+0000 7febd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:42.313 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:20:42.313 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 10 librados: watch_flush enter 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 10 librados: watch_flush exit 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 10 client.4520.objecter successfully canceled tick 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 >> v1:172.21.15.191:6800/1552703795 conn(0x7febc006e160 legacy=0x7febc0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 10 monclient: shutdown 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 >> v1:172.21.15.162:6790/0 conn(0x7febdc0a98f0 legacy=0x7febdc14e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:42.316 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 shutdown_connections 2024-03-25T18:20:42.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 >> 172.21.15.162:0/1825103315 conn(0x7febdc0b7ea0 msgr2=0x7febdc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:42.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 shutdown_connections 2024-03-25T18:20:42.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 1 -- 172.21.15.162:0/1825103315 wait complete. 2024-03-25T18:20:42.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:42.314+0000 7febe44c8640 1 librados: shutdown 2024-03-25T18:20:42.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:49.134 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:49.249 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.248+0000 7f4ba58b3640 10 monclient: get_monmap_and_config 2024-03-25T18:20:49.249 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.248+0000 7f4ba58b3640 10 monclient: build_initial_monmap 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.248+0000 7f4ba58b3640 10 monclient: monmap: 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:20:49.249819+0000 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:20:49.249819+0000 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:20:49.250 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:49.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.249+0000 7f4ba58b3640 1 Processor -- start 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.249+0000 7f4ba58b3640 1 -- start start 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.249+0000 7f4ba58b3640 10 monclient: init 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba00b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 10 monclient(hunting): picked mon.noname-a con 0x7f4ba00b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 10 monclient(hunting): picked mon.noname-c con 0x7f4ba014be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00af940 con 0x7f4ba00b1730 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00b0a70 con 0x7f4ba014be90 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00b0d50 con 0x7f4ba00b4d60 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:49.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.250+0000 7f4ba58b3640 10 monclient(hunting): authenticate will time out at 7948.524033s 2024-03-25T18:20:49.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.256+0000 7f4b9e7fc640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f4ba00b1730 0x7f4ba00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:58966/0 (socket says 172.21.15.162:58966) 2024-03-25T18:20:49.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.256+0000 7f4b9e7fc640 1 -- 172.21.15.162:0/1615421435 learned_addr learned my addr 172.21.15.162:0/1615421435 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:49.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.265+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441965611 0 0) 0x7f4ba00af940 con 0x7f4ba00b1730 2024-03-25T18:20:49.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.265+0000 7f4b9dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.265+0000 7f4b9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.265+0000 7f4b9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.265+0000 7f4b9dffb640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:20:49.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.265+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b88003520 con 0x7f4ba00b1730 2024-03-25T18:20:49.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.272+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479030447 0 0) 0x7f4ba00b0d50 con 0x7f4ba00b4d60 2024-03-25T18:20:49.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.272+0000 7f4b9dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.272+0000 7f4b9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.272+0000 7f4b9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.272+0000 7f4b9dffb640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:20:49.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.272+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba00af940 con 0x7f4ba00b4d60 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3899406541 0 0) 0x7f4b88003520 con 0x7f4ba00b1730 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 >> v1:172.21.15.162:6790/0 conn(0x7f4ba00b4d60 legacy=0x7f4ba013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 >> v1:172.21.15.191:6789/0 conn(0x7f4ba014be90 legacy=0x7f4ba014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 1 monclient: found mon.noname-a 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00b1030 con 0x7f4ba00b1730 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:49.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4b94001690 con 0x7f4ba00b1730 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4b9dffb640 10 monclient: dump: 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:49.281 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:49.282 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4ba58b3640 5 monclient: authenticate success, global_id 4536 2024-03-25T18:20:49.282 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.278+0000 7f4ba58b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:49.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4b9dffb640 1 -- 172.21.15.162:0/1615421435 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b940017f0 con 0x7f4ba00b1730 2024-03-25T18:20:49.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4b9dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:49.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:49.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 4 set_mon_vals no callback set 2024-03-25T18:20:49.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 10 monclient: shutdown 2024-03-25T18:20:49.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:49.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 1 -- 172.21.15.162:0/1615421435 >> v1:172.21.15.162:6789/0 conn(0x7f4ba00b1730 legacy=0x7f4ba00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 1 -- 172.21.15.162:0/1615421435 shutdown_connections 2024-03-25T18:20:49.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 1 -- 172.21.15.162:0/1615421435 >> 172.21.15.162:0/1615421435 conn(0x7f4ba00617a0 msgr2=0x7f4ba0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:49.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 1 -- 172.21.15.162:0/1615421435 shutdown_connections 2024-03-25T18:20:49.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.285+0000 7f4ba58b3640 1 -- 172.21.15.162:0/1615421435 wait complete. 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.286+0000 7f4ba58b3640 10 monclient: build_initial_monmap 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.286+0000 7f4ba58b3640 10 monclient: monmap: 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:49.289 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.286+0000 7f4ba58b3640 1 librados: starting msgr at 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.286+0000 7f4ba58b3640 1 librados: starting objecter 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.286+0000 7f4ba58b3640 1 Processor -- start 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.287+0000 7f4ba58b3640 1 -- start start 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.287+0000 7f4ba58b3640 1 librados: setting wanted keys 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.287+0000 7f4ba58b3640 1 librados: calling monclient init 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.287+0000 7f4ba58b3640 10 monclient: init 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.287+0000 7f4ba58b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.287+0000 7f4ba58b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.287+0000 7f4ba58b3640 10 monclient(hunting): picked mon.noname-a con 0x7f4ba00b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.288+0000 7f4ba58b3640 10 monclient(hunting): picked mon.noname-c con 0x7f4ba00b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.288+0000 7f4ba58b3640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba014be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.288+0000 7f4ba58b3640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba01376f0 con 0x7f4ba00b1730 2024-03-25T18:20:49.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.288+0000 7f4ba58b3640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba013a540 con 0x7f4ba014be90 2024-03-25T18:20:49.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.288+0000 7f4ba58b3640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00aa8d0 con 0x7f4ba00b4d60 2024-03-25T18:20:49.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.288+0000 7f4ba58b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:49.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.288+0000 7f4ba58b3640 10 monclient(hunting): authenticate will time out at 7948.561814s 2024-03-25T18:20:49.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.302+0000 7f4b9f7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f4ba014be90 0x7f4ba0136fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:48666/0 (socket says 172.21.15.162:48666) 2024-03-25T18:20:49.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.302+0000 7f4b9f7fe640 1 -- 172.21.15.162:0/2830135865 learned_addr learned my addr 172.21.15.162:0/2830135865 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:49.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.304+0000 7f4b87fff640 10 client.?.objecter ms_handle_connect 0x7f4ba00b1730 2024-03-25T18:20:49.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.304+0000 7f4b87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:49.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.305+0000 7f4b87fff640 10 client.?.objecter ms_handle_connect 0x7f4ba014be90 2024-03-25T18:20:49.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.305+0000 7f4b87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:49.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.305+0000 7f4b87fff640 10 client.?.objecter ms_handle_connect 0x7f4ba00b4d60 2024-03-25T18:20:49.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.305+0000 7f4b87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:49.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.310+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100767330 0 0) 0x7f4ba013a540 con 0x7f4ba014be90 2024-03-25T18:20:49.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.310+0000 7f4b87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.310+0000 7f4b87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.310+0000 7f4b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.310+0000 7f4b87fff640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:20:49.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.310+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b8c004eb0 con 0x7f4ba014be90 2024-03-25T18:20:49.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.314+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813246829 0 0) 0x7f4ba01376f0 con 0x7f4ba00b1730 2024-03-25T18:20:49.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.314+0000 7f4b87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:49.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.314+0000 7f4b87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:49.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.314+0000 7f4b87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:49.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.314+0000 7f4b87fff640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:20:49.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.314+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba013a540 con 0x7f4ba00b1730 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.319+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3583807990 0 0) 0x7f4b8c004eb0 con 0x7f4ba014be90 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.319+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ba01376f0 con 0x7f4ba014be90 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.319+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4b98001c90 con 0x7f4ba014be90 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.319+0000 7f4b87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.319+0000 7f4b87fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.319+0000 7f4b87fff640 10 monclient(hunting): dump: 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:49.320 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:49.321 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:49.321 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:49.321 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:49.321 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:49.321 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:49.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.319+0000 7f4b87fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3230294951 0 0) 0x7f4ba01376f0 con 0x7f4ba014be90 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 >> v1:172.21.15.162:6790/0 conn(0x7f4ba00b4d60 legacy=0x7f4ba01368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 >> v1:172.21.15.162:6789/0 conn(0x7f4ba00b1730 legacy=0x7f4ba0139e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 1 monclient: found mon.b 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00aabb0 con 0x7f4ba014be90 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:49.327 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4b87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 10 monclient: _renew_subs 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba01387c0 con 0x7f4ba014be90 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 10 monclient: _renew_subs 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba013b0b0 con 0x7f4ba014be90 2024-03-25T18:20:49.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.325+0000 7f4ba58b3640 1 librados: init done 2024-03-25T18:20:49.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.326+0000 7f4ba58b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:49.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.326+0000 7f4ba58b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:49.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.326+0000 7f4ba58b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:49.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.326+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba00627b0 con 0x7f4ba014be90 2024-03-25T18:20:49.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.614+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b98003990 con 0x7f4ba014be90 2024-03-25T18:20:49.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.614+0000 7f4b87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:49.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.614+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4b98003d60 con 0x7f4ba014be90 2024-03-25T18:20:49.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.614+0000 7f4b87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:49.615 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.614+0000 7f4b87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.614+0000 7f4b87fff640 10 monclient: dump: 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:49.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.614+0000 7f4b9d7fa640 4 set_mon_vals no callback set 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.615+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f4b98003f80 con 0x7f4ba014be90 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.616+0000 7f4b9effd640 10 monclient: get_auth_request con 0x7f4b8c06dd40 auth_method 0 2024-03-25T18:20:49.617 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.616+0000 7f4b87fff640 10 client.4522.objecter ms_handle_connect 0x7f4b8c06dd40 2024-03-25T18:20:49.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.620+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f4b98054a90 con 0x7f4ba014be90 2024-03-25T18:20:49.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.620+0000 7f4b87fff640 10 client.4522.objecter ms_dispatch 0x7f4ba0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:49.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.620+0000 7f4b87fff640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:49.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.620+0000 7f4b87fff640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:49.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.620+0000 7f4b87fff640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:20:49.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.620+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b980505a0 con 0x7f4ba014be90 2024-03-25T18:20:49.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.620+0000 7f4b87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:49.621 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.620+0000 7f4b87fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:49.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.863+0000 7f4ba58b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:49.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.863+0000 7f4ba58b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:49.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.863+0000 7f4ba58b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:20:49.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.863+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ba0069f40 con 0x7f4ba014be90 2024-03-25T18:20:49.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.893+0000 7f4b87fff640 1 -- 172.21.15.162:0/2830135865 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f4b980524d0 con 0x7f4ba014be90 2024-03-25T18:20:49.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.893+0000 7f4b87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:49.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.893+0000 7f4b87fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:49.894 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:20:49.894 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:49.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 10 librados: watch_flush enter 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 10 librados: watch_flush exit 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 >> v1:172.21.15.191:6800/1552703795 conn(0x7f4b8c06dd40 legacy=0x7f4b8c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 10 monclient: shutdown 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 >> v1:172.21.15.191:6789/0 conn(0x7f4ba014be90 legacy=0x7f4ba0136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 shutdown_connections 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 >> 172.21.15.162:0/2830135865 conn(0x7f4ba00617a0 msgr2=0x7f4ba0145290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 shutdown_connections 2024-03-25T18:20:49.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 1 -- 172.21.15.162:0/2830135865 wait complete. 2024-03-25T18:20:49.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:49.896+0000 7f4ba58b3640 1 librados: shutdown 2024-03-25T18:20:49.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:56.715 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:56.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.829+0000 7fa0e7217640 10 monclient: get_monmap_and_config 2024-03-25T18:20:56.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.829+0000 7fa0e7217640 10 monclient: build_initial_monmap 2024-03-25T18:20:56.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.829+0000 7fa0e7217640 10 monclient: monmap: 2024-03-25T18:20:56.830 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:56.830 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:56.830 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:20:56.830540+0000 2024-03-25T18:20:56.831 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:20:56.830540+0000 2024-03-25T18:20:56.831 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:56.831 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:56.831 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:56.831 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:20:56.831 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:20:56.831 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 1 Processor -- start 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 1 -- start start 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 10 monclient: init 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 10 monclient(hunting): picked mon.noname-a con 0x7fa0e00a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 10 monclient(hunting): picked mon.noname-b con 0x7fa0e013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 10 monclient(hunting): picked mon.noname-c con 0x7fa0e01430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e0142520 con 0x7fa0e00a98f0 2024-03-25T18:20:56.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e00b28a0 con 0x7fa0e01430c0 2024-03-25T18:20:56.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e00b2a30 con 0x7fa0e013f710 2024-03-25T18:20:56.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.830+0000 7fa0e7217640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:56.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.831+0000 7fa0e7217640 10 monclient(hunting): authenticate will time out at 7956.104654s 2024-03-25T18:20:56.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.835+0000 7fa0dffff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fa0e013f710 0x7fa0e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:60592/0 (socket says 172.21.15.162:60592) 2024-03-25T18:20:56.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.835+0000 7fa0dffff640 1 -- 172.21.15.162:0/110029813 learned_addr learned my addr 172.21.15.162:0/110029813 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:56.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.852+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519931547 0 0) 0x7fa0e00b2a30 con 0x7fa0e013f710 2024-03-25T18:20:56.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.852+0000 7fa0df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.852+0000 7fa0df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.852+0000 7fa0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.852+0000 7fa0df7fe640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:20:56.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.852+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d0003520 con 0x7fa0e013f710 2024-03-25T18:20:56.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.855+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106501138 0 0) 0x7fa0e00b28a0 con 0x7fa0e01430c0 2024-03-25T18:20:56.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.855+0000 7fa0df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.855+0000 7fa0df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.855+0000 7fa0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.855+0000 7fa0df7fe640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:20:56.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.855+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e00b2a30 con 0x7fa0e01430c0 2024-03-25T18:20:56.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.871+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264272340 0 0) 0x7fa0e0142520 con 0x7fa0e00a98f0 2024-03-25T18:20:56.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.871+0000 7fa0df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.871+0000 7fa0df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.871+0000 7fa0df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.871+0000 7fa0df7fe640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:20:56.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.871+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e00b28a0 con 0x7fa0e00a98f0 2024-03-25T18:20:56.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4192130678 0 0) 0x7fa0d0003520 con 0x7fa0e013f710 2024-03-25T18:20:56.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 >> v1:172.21.15.191:6789/0 conn(0x7fa0e01430c0 legacy=0x7fa0e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 >> v1:172.21.15.162:6789/0 conn(0x7fa0e00a98f0 legacy=0x7fa0e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:56.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 1 monclient: found mon.noname-b 2024-03-25T18:20:56.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:56.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e00b44c0 con 0x7fa0e013f710 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa0cc001690 con 0x7fa0e013f710 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0df7fe640 10 monclient: dump: 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:56.880 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:56.881 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:56.881 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0e7217640 5 monclient: authenticate success, global_id 4529 2024-03-25T18:20:56.881 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.877+0000 7fa0e7217640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:56.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.887+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0cc0017f0 con 0x7fa0e013f710 2024-03-25T18:20:56.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.887+0000 7fa0df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:56.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.887+0000 7fa0df7fe640 1 -- 172.21.15.162:0/110029813 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa0cc001c00 con 0x7fa0e013f710 2024-03-25T18:20:56.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.887+0000 7fa0df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.887+0000 7fa0df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:56.888 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.887+0000 7fa0df7fe640 10 monclient: dump: 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.887+0000 7fa0e7217640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:56.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.887+0000 7fa0e7217640 4 set_mon_vals no callback set 2024-03-25T18:20:56.890 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.888+0000 7fa0e7217640 10 monclient: shutdown 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.888+0000 7fa0e7217640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.888+0000 7fa0e7217640 1 -- 172.21.15.162:0/110029813 >> v1:172.21.15.162:6790/0 conn(0x7fa0e013f710 legacy=0x7fa0e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.888+0000 7fa0e7217640 1 -- 172.21.15.162:0/110029813 shutdown_connections 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.888+0000 7fa0e7217640 1 -- 172.21.15.162:0/110029813 >> 172.21.15.162:0/110029813 conn(0x7fa0e00b7ea0 msgr2=0x7fa0e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.888+0000 7fa0e7217640 1 -- 172.21.15.162:0/110029813 shutdown_connections 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.888+0000 7fa0e7217640 1 -- 172.21.15.162:0/110029813 wait complete. 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 10 monclient: build_initial_monmap 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 10 monclient: monmap: 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:20:56.891 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 1 librados: starting msgr at 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 1 librados: starting objecter 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 1 Processor -- start 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 1 -- start start 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 1 librados: setting wanted keys 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 1 librados: calling monclient init 2024-03-25T18:20:56.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.889+0000 7fa0e7217640 10 monclient: init 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.890+0000 7fa0e7217640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.890+0000 7fa0e7217640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.890+0000 7fa0e7217640 10 monclient(hunting): picked mon.noname-c con 0x7fa0e00a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.890+0000 7fa0e7217640 10 monclient(hunting): picked mon.noname-b con 0x7fa0e013f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.891+0000 7fa0e7217640 10 monclient(hunting): picked mon.noname-a con 0x7fa0e01430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.891+0000 7fa0e7217640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e00ab8b0 con 0x7fa0e01430c0 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.891+0000 7fa0e7217640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e00abb90 con 0x7fa0e013f710 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.891+0000 7fa0e7217640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e0137870 con 0x7fa0e00a98f0 2024-03-25T18:20:56.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.891+0000 7fa0e7217640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:56.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.891+0000 7fa0e7217640 10 monclient(hunting): authenticate will time out at 7956.164770s 2024-03-25T18:20:56.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.899+0000 7fa0e4fb3640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fa0e00a98f0 0x7fa0e01350d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:60648/0 (socket says 172.21.15.162:60648) 2024-03-25T18:20:56.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.899+0000 7fa0e4fb3640 1 -- 172.21.15.162:0/1175358468 learned_addr learned my addr 172.21.15.162:0/1175358468 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:20:56.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.900+0000 7fa0dd7fa640 10 client.?.objecter ms_handle_connect 0x7fa0e00a98f0 2024-03-25T18:20:56.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.900+0000 7fa0dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:56.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.902+0000 7fa0dd7fa640 10 client.?.objecter ms_handle_connect 0x7fa0e01430c0 2024-03-25T18:20:56.903 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.902+0000 7fa0dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:56.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.911+0000 7fa0dd7fa640 10 client.?.objecter ms_handle_connect 0x7fa0e013f710 2024-03-25T18:20:56.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.911+0000 7fa0dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:56.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.916+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710009329 0 0) 0x7fa0e0137870 con 0x7fa0e00a98f0 2024-03-25T18:20:56.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.916+0000 7fa0dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.917 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.916+0000 7fa0dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.916+0000 7fa0dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.916+0000 7fa0dd7fa640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:20:56.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.916+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c4004eb0 con 0x7fa0e00a98f0 2024-03-25T18:20:56.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.919+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525703051 0 0) 0x7fa0e00ab8b0 con 0x7fa0e01430c0 2024-03-25T18:20:56.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.919+0000 7fa0dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.919+0000 7fa0dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.919+0000 7fa0dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.919+0000 7fa0dd7fa640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:20:56.921 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.919+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e0137870 con 0x7fa0e01430c0 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.928+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1113952864 0 0) 0x7fa0c4004eb0 con 0x7fa0e00a98f0 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.928+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0e00ab8b0 con 0x7fa0e00a98f0 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.928+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa0d4001a10 con 0x7fa0e00a98f0 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.928+0000 7fa0dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.928+0000 7fa0dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.928+0000 7fa0dd7fa640 10 monclient(hunting): dump: 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:56.929 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.928+0000 7fa0dd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.929+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047295515 0 0) 0x7fa0e00abb90 con 0x7fa0e013f710 2024-03-25T18:20:56.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.929+0000 7fa0dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.929+0000 7fa0dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.929+0000 7fa0dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.929+0000 7fa0dd7fa640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:20:56.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.929+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0c4004eb0 con 0x7fa0e013f710 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.942+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3638873712 0 0) 0x7fa0e0137870 con 0x7fa0e01430c0 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.942+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0e00abb90 con 0x7fa0e01430c0 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.943+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa0d8001b50 con 0x7fa0e01430c0 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.943+0000 7fa0dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.943+0000 7fa0dd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.943+0000 7fa0dd7fa640 10 monclient(hunting): dump: 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:56.944 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:56.945 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.945 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:56.945 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:56.945 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:56.945 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:56.945 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:56.949 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3314067110 0 0) 0x7fa0e00ab8b0 con 0x7fa0e00a98f0 2024-03-25T18:20:56.949 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 >> v1:172.21.15.191:6789/0 conn(0x7fa0e013f710 legacy=0x7fa0e01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.949 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 >> v1:172.21.15.162:6789/0 conn(0x7fa0e01430c0 legacy=0x7fa0e01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.949 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 1 monclient: found mon.c 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e01389e0 con 0x7fa0e00a98f0 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 5 monclient: authenticate success, global_id 4532 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 10 monclient: _renew_subs 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e0136d90 con 0x7fa0e00a98f0 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 10 monclient: _renew_subs 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e01384c0 con 0x7fa0e00a98f0 2024-03-25T18:20:56.950 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.948+0000 7fa0e7217640 1 librados: init done 2024-03-25T18:20:56.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.949+0000 7fa0e7217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.949+0000 7fa0e7217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.949+0000 7fa0e7217640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:56.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:56.949+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e0136f80 con 0x7fa0e00a98f0 2024-03-25T18:20:57.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.259+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0d4001c70 con 0x7fa0e00a98f0 2024-03-25T18:20:57.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.259+0000 7fa0dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:57.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.259+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa0d4003a60 con 0x7fa0e00a98f0 2024-03-25T18:20:57.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.259+0000 7fa0dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:57.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.259+0000 7fa0dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.259+0000 7fa0dd7fa640 10 monclient: dump: 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:20:57.261 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.259+0000 7fa0deffd640 4 set_mon_vals no callback set 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.260+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fa0d4003c80 con 0x7fa0e00a98f0 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.261+0000 7fa0dffff640 10 monclient: get_auth_request con 0x7fa0c406dfb0 auth_method 0 2024-03-25T18:20:57.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.261+0000 7fa0dd7fa640 10 client.4532.objecter ms_handle_connect 0x7fa0c406dfb0 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.266+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fa0d4054880 con 0x7fa0e00a98f0 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.266+0000 7fa0dd7fa640 10 client.4532.objecter ms_dispatch 0x7fa0e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.266+0000 7fa0dd7fa640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.266+0000 7fa0dd7fa640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.266+0000 7fa0dd7fa640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.266+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0d4050390 con 0x7fa0e00a98f0 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.266+0000 7fa0dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:57.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.266+0000 7fa0dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:57.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.526+0000 7fa0e7217640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:57.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.526+0000 7fa0e7217640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:57.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.526+0000 7fa0e7217640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:20:57.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.526+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0e00627b0 con 0x7fa0e00a98f0 2024-03-25T18:20:57.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.552+0000 7fa0dd7fa640 1 -- 172.21.15.162:0/1175358468 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fa0d40522c0 con 0x7fa0e00a98f0 2024-03-25T18:20:57.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.552+0000 7fa0dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:57.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.552+0000 7fa0dd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:57.554 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:20:57.554 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:57.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.554+0000 7fa0e7217640 10 librados: watch_flush enter 2024-03-25T18:20:57.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.554+0000 7fa0e7217640 10 librados: watch_flush exit 2024-03-25T18:20:57.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.554+0000 7fa0e7217640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T18:20:57.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.554+0000 7fa0e7217640 10 client.4532.objecter successfully canceled tick 2024-03-25T18:20:57.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 >> v1:172.21.15.191:6800/1552703795 conn(0x7fa0c406dfb0 legacy=0x7fa0c4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 10 monclient: shutdown 2024-03-25T18:20:57.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:57.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 >> v1:172.21.15.162:6790/0 conn(0x7fa0e00a98f0 legacy=0x7fa0e01350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:57.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 shutdown_connections 2024-03-25T18:20:57.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 >> 172.21.15.162:0/1175358468 conn(0x7fa0e00b7ea0 msgr2=0x7fa0e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:57.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 shutdown_connections 2024-03-25T18:20:57.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 1 -- 172.21.15.162:0/1175358468 wait complete. 2024-03-25T18:20:57.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:20:57.555+0000 7fa0e7217640 1 librados: shutdown 2024-03-25T18:20:57.567 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:58.077 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:10.305 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:58.132 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:10.359 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:58.166 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:10.393 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:58.192 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:10.420 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:58.196 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:10.412 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:58.197 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:10.425 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:58.207 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:10.423 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:58.225 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:10.452 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:58.228 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:10.444 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:04.373 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:04.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.489+0000 7fa669ce6640 10 monclient: get_monmap_and_config 2024-03-25T18:21:04.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.489+0000 7fa669ce6640 10 monclient: build_initial_monmap 2024-03-25T18:21:04.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.489+0000 7fa669ce6640 10 monclient: monmap: 2024-03-25T18:21:04.490 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:04.490 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:04.490 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:21:04.490156+0000 2024-03-25T18:21:04.490 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:21:04.490156+0000 2024-03-25T18:21:04.491 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:04.491 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:04.491 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:04.491 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:21:04.491 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:21:04.491 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:04.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.490+0000 7fa669ce6640 1 Processor -- start 2024-03-25T18:21:04.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.490+0000 7fa669ce6640 1 -- start start 2024-03-25T18:21:04.492 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.490+0000 7fa669ce6640 10 monclient: init 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 10 monclient(hunting): picked mon.noname-b con 0x7fa6640a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 10 monclient(hunting): picked mon.noname-a con 0x7fa66413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 10 monclient(hunting): picked mon.noname-c con 0x7fa6641430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa664142520 con 0x7fa66413f710 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6640b28a0 con 0x7fa6641430c0 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6640b2a30 con 0x7fa6640a98f0 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:04.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.491+0000 7fa669ce6640 10 monclient(hunting): authenticate will time out at 7963.765220s 2024-03-25T18:21:04.511 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.510+0000 7fa663fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa6641430c0 0x7fa66414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:42130/0 (socket says 172.21.15.162:42130) 2024-03-25T18:21:04.512 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.510+0000 7fa663fff640 1 -- 172.21.15.162:0/2305900986 learned_addr learned my addr 172.21.15.162:0/2305900986 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:04.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.528+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852685970 0 0) 0x7fa6640b28a0 con 0x7fa6641430c0 2024-03-25T18:21:04.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.528+0000 7fa6627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.528+0000 7fa6627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.528+0000 7fa6627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.528+0000 7fa6627fc640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:21:04.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.528+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa64c003520 con 0x7fa6641430c0 2024-03-25T18:21:04.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.543+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455066823 0 0) 0x7fa6640b2a30 con 0x7fa6640a98f0 2024-03-25T18:21:04.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.543+0000 7fa6627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.543+0000 7fa6627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.543+0000 7fa6627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.543+0000 7fa6627fc640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:21:04.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.543+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6640b28a0 con 0x7fa6640a98f0 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4083453990 0 0) 0x7fa64c003520 con 0x7fa6641430c0 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 >> v1:172.21.15.162:6790/0 conn(0x7fa6640a98f0 legacy=0x7fa6640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 >> v1:172.21.15.162:6789/0 conn(0x7fa66413f710 legacy=0x7fa6641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 1 monclient: found mon.noname-c 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6640b44c0 con 0x7fa6641430c0 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa65c001a10 con 0x7fa6641430c0 2024-03-25T18:21:04.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa6627fc640 10 monclient: dump: 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa669ce6640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:21:04.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.545+0000 7fa669ce6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.552+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa65c001b70 con 0x7fa6641430c0 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.552+0000 7fa6627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.552+0000 7fa6627fc640 1 -- 172.21.15.162:0/2305900986 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa65c001f80 con 0x7fa6641430c0 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.552+0000 7fa669ce6640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.552+0000 7fa669ce6640 4 set_mon_vals no callback set 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.552+0000 7fa6627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.552+0000 7fa6627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.552+0000 7fa6627fc640 10 monclient: dump: 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:04.554 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.555 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.555 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.555 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:04.555 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:04.555 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:04.555 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:04.555 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:04.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.553+0000 7fa669ce6640 10 monclient: shutdown 2024-03-25T18:21:04.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.553+0000 7fa669ce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:04.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.553+0000 7fa669ce6640 1 -- 172.21.15.162:0/2305900986 >> v1:172.21.15.191:6789/0 conn(0x7fa6641430c0 legacy=0x7fa66414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.554+0000 7fa669ce6640 1 -- 172.21.15.162:0/2305900986 shutdown_connections 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.554+0000 7fa669ce6640 1 -- 172.21.15.162:0/2305900986 >> 172.21.15.162:0/2305900986 conn(0x7fa6640b7ea0 msgr2=0x7fa6640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.554+0000 7fa669ce6640 1 -- 172.21.15.162:0/2305900986 shutdown_connections 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.554+0000 7fa669ce6640 1 -- 172.21.15.162:0/2305900986 wait complete. 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.554+0000 7fa669ce6640 10 monclient: build_initial_monmap 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.554+0000 7fa669ce6640 10 monclient: monmap: 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:04.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.555+0000 7fa669ce6640 1 librados: starting msgr at 2024-03-25T18:21:04.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.555+0000 7fa669ce6640 1 librados: starting objecter 2024-03-25T18:21:04.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.555+0000 7fa669ce6640 1 Processor -- start 2024-03-25T18:21:04.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.555+0000 7fa669ce6640 1 -- start start 2024-03-25T18:21:04.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.555+0000 7fa669ce6640 1 librados: setting wanted keys 2024-03-25T18:21:04.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.555+0000 7fa669ce6640 1 librados: calling monclient init 2024-03-25T18:21:04.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.555+0000 7fa669ce6640 10 monclient: init 2024-03-25T18:21:04.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:04.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:04.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 10 monclient(hunting): picked mon.noname-c con 0x7fa6640a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:04.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 10 monclient(hunting): picked mon.noname-a con 0x7fa66413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:04.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 10 monclient(hunting): picked mon.noname-b con 0x7fa6641430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:04.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa664136ec0 con 0x7fa66413f710 2024-03-25T18:21:04.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa664136c20 con 0x7fa6641430c0 2024-03-25T18:21:04.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa664137db0 con 0x7fa6640a98f0 2024-03-25T18:21:04.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:04.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.556+0000 7fa669ce6640 10 monclient(hunting): authenticate will time out at 7963.830179s 2024-03-25T18:21:04.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.564+0000 7fa6637fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fa6640a98f0 0x7fa6641376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:49116/0 (socket says 172.21.15.162:49116) 2024-03-25T18:21:04.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.564+0000 7fa6637fe640 1 -- 172.21.15.162:0/353725473 learned_addr learned my addr 172.21.15.162:0/353725473 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:04.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.566+0000 7fa64bfff640 10 client.?.objecter ms_handle_connect 0x7fa6640a98f0 2024-03-25T18:21:04.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.566+0000 7fa64bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:04.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.566+0000 7fa64bfff640 10 client.?.objecter ms_handle_connect 0x7fa6641430c0 2024-03-25T18:21:04.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.566+0000 7fa64bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:04.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.623+0000 7fa64bfff640 10 client.?.objecter ms_handle_connect 0x7fa66413f710 2024-03-25T18:21:04.624 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.623+0000 7fa64bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:04.631 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.630+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696789394 0 0) 0x7fa664136c20 con 0x7fa6641430c0 2024-03-25T18:21:04.631 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.630+0000 7fa64bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.631 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.630+0000 7fa64bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.631 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.630+0000 7fa64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.631 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.630+0000 7fa64bfff640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:21:04.631 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.630+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa650004eb0 con 0x7fa6641430c0 2024-03-25T18:21:04.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.639+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323843240 0 0) 0x7fa664136ec0 con 0x7fa66413f710 2024-03-25T18:21:04.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.639+0000 7fa64bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:04.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.639+0000 7fa64bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:04.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.639+0000 7fa64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:04.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.639+0000 7fa64bfff640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:21:04.640 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.639+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa664136c20 con 0x7fa66413f710 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.649+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3804009852 0 0) 0x7fa650004eb0 con 0x7fa6641430c0 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.650+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa664136ec0 con 0x7fa6641430c0 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.650+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa65c001ec0 con 0x7fa6641430c0 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.650+0000 7fa64bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.650+0000 7fa64bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.650+0000 7fa64bfff640 10 monclient(hunting): dump: 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:04.651 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:04.652 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:04.652 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:04.652 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:04.652 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.650+0000 7fa64bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.661+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2441812492 0 0) 0x7fa664136c20 con 0x7fa66413f710 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.662+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa650004eb0 con 0x7fa66413f710 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.662+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa6580018a0 con 0x7fa66413f710 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.662+0000 7fa64bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.662+0000 7fa64bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.662+0000 7fa64bfff640 10 monclient(hunting): dump: 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:04.663 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:04.664 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:04.664 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:04.664 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:04.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.665+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4217976351 0 0) 0x7fa664136ec0 con 0x7fa6641430c0 2024-03-25T18:21:04.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 >> v1:172.21.15.162:6790/0 conn(0x7fa6640a98f0 legacy=0x7fa6641376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 >> v1:172.21.15.162:6789/0 conn(0x7fa66413f710 legacy=0x7fa6640b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:04.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 1 monclient: found mon.b 2024-03-25T18:21:04.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa664138b50 con 0x7fa6641430c0 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 10 monclient: _finish_auth 0 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa64bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 10 monclient: _renew_subs 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa664139830 con 0x7fa6641430c0 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 10 monclient: _renew_subs 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6641369b0 con 0x7fa6641430c0 2024-03-25T18:21:04.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.666+0000 7fa669ce6640 1 librados: init done 2024-03-25T18:21:04.669 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.667+0000 7fa669ce6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:04.669 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.667+0000 7fa669ce6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:04.669 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.667+0000 7fa669ce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:04.669 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.667+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6641373c0 con 0x7fa6641430c0 2024-03-25T18:21:04.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.938+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa65c0020e0 con 0x7fa6641430c0 2024-03-25T18:21:04.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.938+0000 7fa64bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:04.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.938+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa65c004e10 con 0x7fa6641430c0 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.938+0000 7fa64bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.938+0000 7fa64bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.938+0000 7fa64bfff640 10 monclient: dump: 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:04.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.938+0000 7fa661ffb640 4 set_mon_vals no callback set 2024-03-25T18:21:04.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.939+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fa65c01ade0 con 0x7fa6641430c0 2024-03-25T18:21:04.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.939+0000 7fa6637fe640 10 monclient: get_auth_request con 0x7fa65006de90 auth_method 0 2024-03-25T18:21:04.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.940+0000 7fa64bfff640 10 client.4534.objecter ms_handle_connect 0x7fa65006de90 2024-03-25T18:21:04.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.944+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fa65c054a00 con 0x7fa6641430c0 2024-03-25T18:21:04.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.944+0000 7fa64bfff640 10 client.4534.objecter ms_dispatch 0x7fa664000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:04.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.944+0000 7fa64bfff640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:04.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.944+0000 7fa64bfff640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:04.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.944+0000 7fa64bfff640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T18:21:04.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.944+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa65c050340 con 0x7fa6641430c0 2024-03-25T18:21:04.945 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.944+0000 7fa64bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:04.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:04.944+0000 7fa64bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:05.196 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.195+0000 7fa669ce6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:05.196 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.195+0000 7fa669ce6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:05.196 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.195+0000 7fa669ce6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:05.196 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.195+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa66414f320 con 0x7fa6641430c0 2024-03-25T18:21:05.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.215+0000 7fa64bfff640 1 -- 172.21.15.162:0/353725473 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fa65c052440 con 0x7fa6641430c0 2024-03-25T18:21:05.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.215+0000 7fa64bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:05.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.215+0000 7fa64bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:05.217 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:21:05.217 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:05.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.217+0000 7fa669ce6640 10 librados: watch_flush enter 2024-03-25T18:21:05.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 10 librados: watch_flush exit 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 10 client.4534.objecter successfully canceled tick 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 >> v1:172.21.15.191:6800/1552703795 conn(0x7fa65006de90 legacy=0x7fa650070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 10 monclient: shutdown 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 >> v1:172.21.15.191:6789/0 conn(0x7fa6641430c0 legacy=0x7fa6641e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 shutdown_connections 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 >> 172.21.15.162:0/353725473 conn(0x7fa6640b7ea0 msgr2=0x7fa66413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 shutdown_connections 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 1 -- 172.21.15.162:0/353725473 wait complete. 2024-03-25T18:21:05.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:05.218+0000 7fa669ce6640 1 librados: shutdown 2024-03-25T18:21:05.231 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:12.037 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:12.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.152+0000 7f90605cf640 10 monclient: get_monmap_and_config 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.152+0000 7f90605cf640 10 monclient: build_initial_monmap 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.152+0000 7f90605cf640 10 monclient: monmap: 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:21:12.153994+0000 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:21:12.153994+0000 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:21:12.154 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:12.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.153+0000 7f90605cf640 1 Processor -- start 2024-03-25T18:21:12.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.153+0000 7f90605cf640 1 -- start start 2024-03-25T18:21:12.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.153+0000 7f90605cf640 10 monclient: init 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 10 monclient(hunting): picked mon.noname-b con 0x7f90580b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 10 monclient(hunting): picked mon.noname-c con 0x7f90580b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 10 monclient(hunting): picked mon.noname-a con 0x7f905814be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90580af940 con 0x7f905814be90 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90580b0a70 con 0x7f90580b1730 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90580b0d50 con 0x7f90580b4d60 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:12.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.154+0000 7f90605cf640 10 monclient(hunting): authenticate will time out at 7971.428373s 2024-03-25T18:21:12.171 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.170+0000 7f905db6a640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f90580b1730 0x7f90580b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:47020/0 (socket says 172.21.15.162:47020) 2024-03-25T18:21:12.171 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.170+0000 7f905db6a640 1 -- 172.21.15.162:0/961877915 learned_addr learned my addr 172.21.15.162:0/961877915 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:12.179 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.178+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125467104 0 0) 0x7f90580af940 con 0x7f905814be90 2024-03-25T18:21:12.179 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.178+0000 7f905d369640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.178+0000 7f905d369640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.178+0000 7f905d369640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.178+0000 7f905d369640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:21:12.180 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.178+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9044003520 con 0x7f905814be90 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3895206304 0 0) 0x7f9044003520 con 0x7f905814be90 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 >> v1:172.21.15.162:6790/0 conn(0x7f90580b4d60 legacy=0x7f905813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 >> v1:172.21.15.191:6789/0 conn(0x7f90580b1730 legacy=0x7f90580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 10 monclient: _finish_hunting 0 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 1 monclient: found mon.noname-a 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90580b1030 con 0x7f905814be90 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 10 monclient: _finish_auth 0 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 10 monclient: _check_auth_tickets 2024-03-25T18:21:12.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9050001a10 con 0x7f905814be90 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f905d369640 10 monclient: dump: 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:12.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f90605cf640 5 monclient: authenticate success, global_id 4554 2024-03-25T18:21:12.196 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.192+0000 7f90605cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:12.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.222+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9050001b70 con 0x7f905814be90 2024-03-25T18:21:12.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.222+0000 7f905d369640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.222+0000 7f905d369640 1 -- 172.21.15.162:0/961877915 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9050001f80 con 0x7f905814be90 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.222+0000 7f905d369640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.222+0000 7f905d369640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.222+0000 7f905d369640 10 monclient: dump: 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.222+0000 7f90605cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:12.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.222+0000 7f90605cf640 4 set_mon_vals no callback set 2024-03-25T18:21:12.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.223+0000 7f90605cf640 10 monclient: shutdown 2024-03-25T18:21:12.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.223+0000 7f90605cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:12.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.223+0000 7f90605cf640 1 -- 172.21.15.162:0/961877915 >> v1:172.21.15.162:6789/0 conn(0x7f905814be90 legacy=0x7f905814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.223+0000 7f90605cf640 1 -- 172.21.15.162:0/961877915 shutdown_connections 2024-03-25T18:21:12.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.223+0000 7f90605cf640 1 -- 172.21.15.162:0/961877915 >> 172.21.15.162:0/961877915 conn(0x7f90580617a0 msgr2=0x7f9058061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:12.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.224+0000 7f90605cf640 1 -- 172.21.15.162:0/961877915 shutdown_connections 2024-03-25T18:21:12.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.224+0000 7f90605cf640 1 -- 172.21.15.162:0/961877915 wait complete. 2024-03-25T18:21:12.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.224+0000 7f90605cf640 10 monclient: build_initial_monmap 2024-03-25T18:21:12.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.224+0000 7f90605cf640 10 monclient: monmap: 2024-03-25T18:21:12.226 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:12.226 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:12.226 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.225+0000 7f90605cf640 1 librados: starting msgr at 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.225+0000 7f90605cf640 1 librados: starting objecter 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.225+0000 7f90605cf640 1 Processor -- start 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.225+0000 7f90605cf640 1 -- start start 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.225+0000 7f90605cf640 1 librados: setting wanted keys 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.225+0000 7f90605cf640 1 librados: calling monclient init 2024-03-25T18:21:12.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.225+0000 7f90605cf640 10 monclient: init 2024-03-25T18:21:12.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:12.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:12.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 10 monclient(hunting): picked mon.noname-a con 0x7f90580b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:12.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 10 monclient(hunting): picked mon.noname-c con 0x7f90580b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:12.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 10 monclient(hunting): picked mon.noname-b con 0x7f905814be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:12.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9058134f90 con 0x7f90580b1730 2024-03-25T18:21:12.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90581414e0 con 0x7f905814be90 2024-03-25T18:21:12.228 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90581417c0 con 0x7f90580b4d60 2024-03-25T18:21:12.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:12.229 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.226+0000 7f90605cf640 10 monclient(hunting): authenticate will time out at 7971.500138s 2024-03-25T18:21:12.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.235+0000 7f905eb6c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f905814be90 0x7f9058134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:47074/0 (socket says 172.21.15.162:47074) 2024-03-25T18:21:12.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.235+0000 7f905eb6c640 1 -- 172.21.15.162:0/2930566021 learned_addr learned my addr 172.21.15.162:0/2930566021 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:12.238 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.237+0000 7f904effd640 10 client.?.objecter ms_handle_connect 0x7f905814be90 2024-03-25T18:21:12.238 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.237+0000 7f904effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:12.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.242+0000 7f904effd640 10 client.?.objecter ms_handle_connect 0x7f90580b4d60 2024-03-25T18:21:12.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.242+0000 7f904effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:12.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.251+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616305804 0 0) 0x7f90581414e0 con 0x7f905814be90 2024-03-25T18:21:12.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.251+0000 7f904effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.251+0000 7f904effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.251+0000 7f904effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.251+0000 7f904effd640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:21:12.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.251+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f903c004eb0 con 0x7f905814be90 2024-03-25T18:21:12.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.253+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89635781 0 0) 0x7f90581417c0 con 0x7f90580b4d60 2024-03-25T18:21:12.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.253+0000 7f904effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.253+0000 7f904effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.253+0000 7f904effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.253+0000 7f904effd640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:21:12.254 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.253+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90581414e0 con 0x7f90580b4d60 2024-03-25T18:21:12.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.257+0000 7f904effd640 10 client.?.objecter ms_handle_connect 0x7f90580b1730 2024-03-25T18:21:12.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.257+0000 7f904effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:12.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.272+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3020553414 0 0) 0x7f90581414e0 con 0x7f90580b4d60 2024-03-25T18:21:12.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.272+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90581417c0 con 0x7f90580b4d60 2024-03-25T18:21:12.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.272+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f90480018a0 con 0x7f90580b4d60 2024-03-25T18:21:12.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.272+0000 7f904effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.272+0000 7f904effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:12.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.272+0000 7f904effd640 10 monclient(hunting): dump: 2024-03-25T18:21:12.273 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:12.274 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.272+0000 7f904effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.275+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2158932475 0 0) 0x7f903c004eb0 con 0x7f905814be90 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.275+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90581414e0 con 0x7f905814be90 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.275+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9050001f40 con 0x7f905814be90 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.275+0000 7f904effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.275+0000 7f904effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.275+0000 7f904effd640 10 monclient(hunting): dump: 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:12.277 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:12.278 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:12.278 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:12.278 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:12.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.288+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989975196 0 0) 0x7f9058134f90 con 0x7f90580b1730 2024-03-25T18:21:12.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:12.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:12.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:12.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:21:12.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f903c004eb0 con 0x7f90580b1730 2024-03-25T18:21:12.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3504008344 0 0) 0x7f90581417c0 con 0x7f90580b4d60 2024-03-25T18:21:12.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 >> v1:172.21.15.191:6789/0 conn(0x7f905814be90 legacy=0x7f9058134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 >> v1:172.21.15.162:6789/0 conn(0x7f90580b1730 legacy=0x7f905813b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 1 monclient: found mon.c 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905813b180 con 0x7f90580b4d60 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 10 monclient: _finish_auth 0 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f904effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 5 monclient: authenticate success, global_id 4544 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 10 monclient: _renew_subs 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9058141be0 con 0x7f90580b4d60 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:12.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 10 monclient: _renew_subs 2024-03-25T18:21:12.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:12.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f905813bfb0 con 0x7f90580b4d60 2024-03-25T18:21:12.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.289+0000 7f90605cf640 1 librados: init done 2024-03-25T18:21:12.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.290+0000 7f90605cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:12.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.290+0000 7f90605cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:12.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.290+0000 7f90605cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:12.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.290+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905813afc0 con 0x7f90580b4d60 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.593+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9048001cd0 con 0x7f90580b4d60 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.593+0000 7f904effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.593+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9048004980 con 0x7f90580b4d60 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.593+0000 7f904effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.593+0000 7f904effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.593+0000 7f904effd640 10 monclient: dump: 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.594 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:12.595 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:12.595 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:12.595 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:12.595 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:12.595 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:12.595 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:12.595 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.593+0000 7f905cb68640 4 set_mon_vals no callback set 2024-03-25T18:21:12.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.593+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f9048004ba0 con 0x7f90580b4d60 2024-03-25T18:21:12.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.594+0000 7f905e36b640 10 monclient: get_auth_request con 0x7f903c06df40 auth_method 0 2024-03-25T18:21:12.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.594+0000 7f904effd640 10 client.4544.objecter ms_handle_connect 0x7f903c06df40 2024-03-25T18:21:12.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.599+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f90480546f0 con 0x7f90580b4d60 2024-03-25T18:21:12.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.599+0000 7f904effd640 10 client.4544.objecter ms_dispatch 0x7f9058000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:12.600 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.599+0000 7f904effd640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:12.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.599+0000 7f904effd640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:12.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.599+0000 7f904effd640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-25T18:21:12.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.599+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9048050200 con 0x7f90580b4d60 2024-03-25T18:21:12.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.599+0000 7f904effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:12.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.599+0000 7f904effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:12.844 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.842+0000 7f90605cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:12.844 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.842+0000 7f90605cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:12.844 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.842+0000 7f90605cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:12.844 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.842+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f905813b310 con 0x7f90580b4d60 2024-03-25T18:21:12.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.991+0000 7f904effd640 1 -- 172.21.15.162:0/2930566021 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f9048052130 con 0x7f90580b4d60 2024-03-25T18:21:12.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.991+0000 7f904effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:12.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.991+0000 7f904effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:12.992 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:21:12.992 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 10 librados: watch_flush enter 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 10 librados: watch_flush exit 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 10 client.4544.objecter successfully canceled tick 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 >> v1:172.21.15.191:6800/1552703795 conn(0x7f903c06df40 legacy=0x7f903c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 10 monclient: shutdown 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 >> v1:172.21.15.162:6790/0 conn(0x7f90580b4d60 legacy=0x7f9058134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.995 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 shutdown_connections 2024-03-25T18:21:12.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 >> 172.21.15.162:0/2930566021 conn(0x7f90580617a0 msgr2=0x7f905813c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:12.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 shutdown_connections 2024-03-25T18:21:12.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 1 -- 172.21.15.162:0/2930566021 wait complete. 2024-03-25T18:21:12.996 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:12.993+0000 7f90605cf640 1 librados: shutdown 2024-03-25T18:21:13.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:18.157 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:30.385 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.211 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:30.438 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.244 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:30.471 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.260 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:30.487 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.265 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:30.481 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.268 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:30.496 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.275 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:30.490 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.296 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:30.524 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:18.306 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:30.522 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:19.813 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:19.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.926+0000 7f2a90225640 10 monclient: get_monmap_and_config 2024-03-25T18:21:19.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.926+0000 7f2a90225640 10 monclient: build_initial_monmap 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.926+0000 7f2a90225640 10 monclient: monmap: 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:21:19.927462+0000 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:21:19.927462+0000 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:21:19.928 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.927+0000 7f2a90225640 1 Processor -- start 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.927+0000 7f2a90225640 1 -- start start 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.927+0000 7f2a90225640 10 monclient: init 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 10 monclient(hunting): picked mon.noname-a con 0x7f2a880b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 10 monclient(hunting): picked mon.noname-c con 0x7f2a880b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 10 monclient(hunting): picked mon.noname-b con 0x7f2a8814be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a880af940 con 0x7f2a880b4d60 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a880b0a70 con 0x7f2a880b1730 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a880b0d50 con 0x7f2a8814be90 2024-03-25T18:21:19.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:19.931 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.928+0000 7f2a90225640 10 monclient(hunting): authenticate will time out at 7979.202037s 2024-03-25T18:21:19.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.934+0000 7f2a8dfc1640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2a880b4d60 0x7f2a8813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:51520/0 (socket says 172.21.15.162:51520) 2024-03-25T18:21:19.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.934+0000 7f2a8dfc1640 1 -- 172.21.15.162:0/2878392031 learned_addr learned my addr 172.21.15.162:0/2878392031 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:19.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.946+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053220515 0 0) 0x7f2a880b0a70 con 0x7f2a880b1730 2024-03-25T18:21:19.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.946+0000 7f2a8cfbf640 10 monclient(hunting): _negotiate 2024-03-25T18:21:19.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.946+0000 7f2a8cfbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:19.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.946+0000 7f2a8cfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:19.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.946+0000 7f2a8cfbf640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:21:19.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.947+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a74003520 con 0x7f2a880b1730 2024-03-25T18:21:19.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.950+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526522674 0 0) 0x7f2a880af940 con 0x7f2a880b4d60 2024-03-25T18:21:19.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.950+0000 7f2a8cfbf640 10 monclient(hunting): _negotiate 2024-03-25T18:21:19.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.950+0000 7f2a8cfbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:19.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.950+0000 7f2a8cfbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:19.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.950+0000 7f2a8cfbf640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:21:19.951 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.950+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a880b0a70 con 0x7f2a880b4d60 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2410457840 0 0) 0x7f2a74003520 con 0x7f2a880b1730 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 >> v1:172.21.15.162:6790/0 conn(0x7f2a8814be90 legacy=0x7f2a8814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 >> v1:172.21.15.162:6789/0 conn(0x7f2a880b4d60 legacy=0x7f2a8813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 10 monclient: _finish_hunting 0 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 1 monclient: found mon.noname-c 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a880b1030 con 0x7f2a880b1730 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 10 monclient: _finish_auth 0 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 10 monclient: _check_auth_tickets 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:19.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2a78001690 con 0x7f2a880b1730 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a8cfbf640 10 monclient: dump: 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a90225640 5 monclient: authenticate success, global_id 4543 2024-03-25T18:21:19.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.957+0000 7f2a90225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:19.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.962+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a780017f0 con 0x7f2a880b1730 2024-03-25T18:21:19.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.962+0000 7f2a8cfbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.962+0000 7f2a8cfbf640 1 -- 172.21.15.162:0/2878392031 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2a78001c00 con 0x7f2a880b1730 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.962+0000 7f2a90225640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.962+0000 7f2a90225640 4 set_mon_vals no callback set 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.962+0000 7f2a8cfbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.962+0000 7f2a8cfbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.962+0000 7f2a8cfbf640 10 monclient: dump: 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:19.964 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.963+0000 7f2a90225640 10 monclient: shutdown 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.963+0000 7f2a90225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.963+0000 7f2a90225640 1 -- 172.21.15.162:0/2878392031 >> v1:172.21.15.191:6789/0 conn(0x7f2a880b1730 legacy=0x7f2a880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.963+0000 7f2a90225640 1 -- 172.21.15.162:0/2878392031 shutdown_connections 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.963+0000 7f2a90225640 1 -- 172.21.15.162:0/2878392031 >> 172.21.15.162:0/2878392031 conn(0x7f2a880617a0 msgr2=0x7f2a88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.963+0000 7f2a90225640 1 -- 172.21.15.162:0/2878392031 shutdown_connections 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.963+0000 7f2a90225640 1 -- 172.21.15.162:0/2878392031 wait complete. 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.964+0000 7f2a90225640 10 monclient: build_initial_monmap 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.964+0000 7f2a90225640 10 monclient: monmap: 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:19.966 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.964+0000 7f2a90225640 1 librados: starting msgr at 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.964+0000 7f2a90225640 1 librados: starting objecter 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.965+0000 7f2a90225640 1 Processor -- start 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.965+0000 7f2a90225640 1 -- start start 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.965+0000 7f2a90225640 1 librados: setting wanted keys 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.965+0000 7f2a90225640 1 librados: calling monclient init 2024-03-25T18:21:19.967 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.965+0000 7f2a90225640 10 monclient: init 2024-03-25T18:21:19.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.965+0000 7f2a90225640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:19.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:19.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 10 monclient(hunting): picked mon.noname-b con 0x7f2a880b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:19.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 10 monclient(hunting): picked mon.noname-c con 0x7f2a880b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:19.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 10 monclient(hunting): picked mon.noname-a con 0x7f2a8814be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:19.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8813b3e0 con 0x7f2a8814be90 2024-03-25T18:21:19.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a881413a0 con 0x7f2a880b1730 2024-03-25T18:21:19.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a88141610 con 0x7f2a880b4d60 2024-03-25T18:21:19.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:19.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.966+0000 7f2a90225640 10 monclient(hunting): authenticate will time out at 7979.240069s 2024-03-25T18:21:19.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.971+0000 7f2a8e7c2640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2a8814be90 0x7f2a881e2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:51534/0 (socket says 172.21.15.162:51534) 2024-03-25T18:21:19.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.971+0000 7f2a8e7c2640 1 -- 172.21.15.162:0/3598565259 learned_addr learned my addr 172.21.15.162:0/3598565259 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:19.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.972+0000 7f2a7e7fc640 10 client.?.objecter ms_handle_connect 0x7f2a8814be90 2024-03-25T18:21:19.973 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.972+0000 7f2a7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:19.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.976+0000 7f2a7e7fc640 10 client.?.objecter ms_handle_connect 0x7f2a880b1730 2024-03-25T18:21:19.977 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.976+0000 7f2a7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:19.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.978+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512640194 0 0) 0x7f2a8813b3e0 con 0x7f2a8814be90 2024-03-25T18:21:19.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.978+0000 7f2a7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:19.979 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.978+0000 7f2a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:19.980 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.978+0000 7f2a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:19.980 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.978+0000 7f2a7e7fc640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:21:19.980 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.978+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a6c004eb0 con 0x7f2a8814be90 2024-03-25T18:21:19.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.980+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015275520 0 0) 0x7f2a881413a0 con 0x7f2a880b1730 2024-03-25T18:21:19.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.980+0000 7f2a7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:19.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.980+0000 7f2a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:19.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.980+0000 7f2a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:19.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.980+0000 7f2a7e7fc640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:21:19.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.980+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8813b3e0 con 0x7f2a880b1730 2024-03-25T18:21:19.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.987+0000 7f2a7e7fc640 10 client.?.objecter ms_handle_connect 0x7f2a880b4d60 2024-03-25T18:21:19.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.987+0000 7f2a7e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.991+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3408119439 0 0) 0x7f2a8813b3e0 con 0x7f2a880b1730 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.991+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a881413a0 con 0x7f2a880b1730 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.991+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2a84001930 con 0x7f2a880b1730 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.991+0000 7f2a7e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.991+0000 7f2a7e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.991+0000 7f2a7e7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:19.992 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:19.993 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:19.993 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:19.993 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:19.993 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:19.993 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:19.993 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.991+0000 7f2a7e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:19.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.997+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330807293 0 0) 0x7f2a88141610 con 0x7f2a880b4d60 2024-03-25T18:21:19.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.997+0000 7f2a7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:19.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.997+0000 7f2a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:19.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.997+0000 7f2a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:19.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.997+0000 7f2a7e7fc640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:21:19.998 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.997+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a8813b3e0 con 0x7f2a880b4d60 2024-03-25T18:21:19.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 857128313 0 0) 0x7f2a881413a0 con 0x7f2a880b1730 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 >> v1:172.21.15.162:6790/0 conn(0x7f2a880b4d60 legacy=0x7f2a88142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 >> v1:172.21.15.162:6789/0 conn(0x7f2a8814be90 legacy=0x7f2a881e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 1 monclient: found mon.b 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a88140380 con 0x7f2a880b1730 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:20.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 10 monclient: _renew_subs 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a88134bf0 con 0x7f2a880b1730 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 10 monclient: _renew_subs 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8813a7e0 con 0x7f2a880b1730 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.998+0000 7f2a90225640 1 librados: init done 2024-03-25T18:21:20.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.999+0000 7f2a90225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:20.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.999+0000 7f2a90225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:20.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.999+0000 7f2a90225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:20.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:19.999+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a880627b0 con 0x7f2a880b1730 2024-03-25T18:21:20.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.275+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a840047e0 con 0x7f2a880b1730 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.275+0000 7f2a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.275+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2a84004af0 con 0x7f2a880b1730 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.275+0000 7f2a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.275+0000 7f2a7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.275+0000 7f2a7e7fc640 10 monclient: dump: 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:20.278 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.275+0000 7f2a7ffff640 4 set_mon_vals no callback set 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.276+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f2a8401aac0 con 0x7f2a880b1730 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.277+0000 7f2a8d7c0640 10 monclient: get_auth_request con 0x7f2a6c06de10 auth_method 0 2024-03-25T18:21:20.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.277+0000 7f2a7e7fc640 10 client.4546.objecter ms_handle_connect 0x7f2a6c06de10 2024-03-25T18:21:20.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.282+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f2a84054800 con 0x7f2a880b1730 2024-03-25T18:21:20.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.282+0000 7f2a7e7fc640 10 client.4546.objecter ms_dispatch 0x7f2a88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:20.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.282+0000 7f2a7e7fc640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:20.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.282+0000 7f2a7e7fc640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:20.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.282+0000 7f2a7e7fc640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:21:20.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.283+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a84050140 con 0x7f2a880b1730 2024-03-25T18:21:20.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.283+0000 7f2a7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:20.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.283+0000 7f2a7e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:20.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.540+0000 7f2a90225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:20.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.540+0000 7f2a90225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:20.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.540+0000 7f2a90225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:20.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.540+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a88140510 con 0x7f2a880b1730 2024-03-25T18:21:20.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.563+0000 7f2a7e7fc640 1 -- 172.21.15.162:0/3598565259 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f2a84052240 con 0x7f2a880b1730 2024-03-25T18:21:20.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.563+0000 7f2a7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:20.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.563+0000 7f2a7e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:20.565 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:21:20.565 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:20.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.565+0000 7f2a90225640 10 librados: watch_flush enter 2024-03-25T18:21:20.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.565+0000 7f2a90225640 10 librados: watch_flush exit 2024-03-25T18:21:20.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.565+0000 7f2a90225640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:21:20.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.565+0000 7f2a90225640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:21:20.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.565+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 >> v1:172.21.15.191:6800/1552703795 conn(0x7f2a6c06de10 legacy=0x7f2a6c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.565+0000 7f2a90225640 10 monclient: shutdown 2024-03-25T18:21:20.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.565+0000 7f2a90225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:20.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.565+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 >> v1:172.21.15.191:6789/0 conn(0x7f2a880b1730 legacy=0x7f2a88141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:20.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.566+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 shutdown_connections 2024-03-25T18:21:20.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.566+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 >> 172.21.15.162:0/3598565259 conn(0x7f2a880617a0 msgr2=0x7f2a880b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:20.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.566+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 shutdown_connections 2024-03-25T18:21:20.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.566+0000 7f2a90225640 1 -- 172.21.15.162:0/3598565259 wait complete. 2024-03-25T18:21:20.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:20.566+0000 7f2a90225640 1 librados: shutdown 2024-03-25T18:21:20.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:27.385 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.502+0000 7f464a16b640 10 monclient: get_monmap_and_config 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.502+0000 7f464a16b640 10 monclient: build_initial_monmap 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.502+0000 7f464a16b640 10 monclient: monmap: 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:21:27.503909+0000 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:21:27.503909+0000 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:27.504 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:21:27.505 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:21:27.505 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:27.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.503+0000 7f464a16b640 1 Processor -- start 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.503+0000 7f464a16b640 1 -- start start 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.503+0000 7f464a16b640 10 monclient: init 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 10 monclient(hunting): picked mon.noname-c con 0x7f46440a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 10 monclient(hunting): picked mon.noname-b con 0x7f464413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 10 monclient(hunting): picked mon.noname-a con 0x7f46441430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4644142520 con 0x7f46441430c0 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440b28a0 con 0x7f46440a98f0 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46440b2a30 con 0x7f464413f710 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:27.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.504+0000 7f464a16b640 10 monclient(hunting): authenticate will time out at 7986.778220s 2024-03-25T18:21:27.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.512+0000 7f4642ffd640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f464413f710 0x7f46441429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:54210/0 (socket says 172.21.15.162:54210) 2024-03-25T18:21:27.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.512+0000 7f4642ffd640 1 -- 172.21.15.162:0/2813301954 learned_addr learned my addr 172.21.15.162:0/2813301954 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:27.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.530+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758597275 0 0) 0x7f46440b2a30 con 0x7f464413f710 2024-03-25T18:21:27.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.530+0000 7f46427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.530+0000 7f46427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.530+0000 7f46427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.530+0000 7f46427fc640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:21:27.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.530+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f462c003520 con 0x7f464413f710 2024-03-25T18:21:27.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.532+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127257380 0 0) 0x7f4644142520 con 0x7f46441430c0 2024-03-25T18:21:27.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.532+0000 7f46427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.532+0000 7f46427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.532+0000 7f46427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.532+0000 7f46427fc640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:21:27.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.532+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46440b2a30 con 0x7f46441430c0 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095496800 0 0) 0x7f462c003520 con 0x7f464413f710 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 >> v1:172.21.15.191:6789/0 conn(0x7f46440a98f0 legacy=0x7f46440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 >> v1:172.21.15.162:6789/0 conn(0x7f46441430c0 legacy=0x7f464414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 1 monclient: found mon.noname-b 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46440b44c0 con 0x7f464413f710 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:27.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4638001690 con 0x7f464413f710 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f464a16b640 5 monclient: authenticate success, global_id 4550 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f464a16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f46427fc640 10 monclient: dump: 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:27.543 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:27.544 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:27.544 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:27.544 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:27.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.540+0000 7f464a16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:27.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.552+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46380017f0 con 0x7f464413f710 2024-03-25T18:21:27.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.552+0000 7f46427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:27.553 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.552+0000 7f46427fc640 1 -- 172.21.15.162:0/2813301954 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4638001c00 con 0x7f464413f710 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.552+0000 7f464a16b640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.552+0000 7f464a16b640 4 set_mon_vals no callback set 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.552+0000 7f46427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.552+0000 7f46427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.552+0000 7f46427fc640 10 monclient: dump: 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:27.554 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:27.555 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.553+0000 7f464a16b640 10 monclient: shutdown 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.553+0000 7f464a16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.553+0000 7f464a16b640 1 -- 172.21.15.162:0/2813301954 >> v1:172.21.15.162:6790/0 conn(0x7f464413f710 legacy=0x7f46441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.553+0000 7f464a16b640 1 -- 172.21.15.162:0/2813301954 shutdown_connections 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.553+0000 7f464a16b640 1 -- 172.21.15.162:0/2813301954 >> 172.21.15.162:0/2813301954 conn(0x7f46440b7ea0 msgr2=0x7f46440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.553+0000 7f464a16b640 1 -- 172.21.15.162:0/2813301954 shutdown_connections 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.553+0000 7f464a16b640 1 -- 172.21.15.162:0/2813301954 wait complete. 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.554+0000 7f464a16b640 10 monclient: build_initial_monmap 2024-03-25T18:21:27.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.554+0000 7f464a16b640 10 monclient: monmap: 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.554+0000 7f464a16b640 1 librados: starting msgr at 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.554+0000 7f464a16b640 1 librados: starting objecter 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.555+0000 7f464a16b640 1 Processor -- start 2024-03-25T18:21:27.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.555+0000 7f464a16b640 1 -- start start 2024-03-25T18:21:27.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.555+0000 7f464a16b640 1 librados: setting wanted keys 2024-03-25T18:21:27.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.555+0000 7f464a16b640 1 librados: calling monclient init 2024-03-25T18:21:27.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.555+0000 7f464a16b640 10 monclient: init 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 10 monclient(hunting): picked mon.noname-a con 0x7f46440a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 10 monclient(hunting): picked mon.noname-b con 0x7f464413f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 10 monclient(hunting): picked mon.noname-c con 0x7f46441430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464413be70 con 0x7f46440a98f0 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4644131660 con 0x7f464413f710 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46441317f0 con 0x7f46441430c0 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:27.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.556+0000 7f464a16b640 10 monclient(hunting): authenticate will time out at 7986.829981s 2024-03-25T18:21:27.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.562+0000 7f4643fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f46441430c0 0x7f46441e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:54256/0 (socket says 172.21.15.162:54256) 2024-03-25T18:21:27.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.562+0000 7f4643fff640 1 -- 172.21.15.162:0/3877522325 learned_addr learned my addr 172.21.15.162:0/3877522325 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:27.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.568+0000 7f462bfff640 10 client.?.objecter ms_handle_connect 0x7f46441430c0 2024-03-25T18:21:27.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.568+0000 7f462bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:27.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.571+0000 7f462bfff640 10 client.?.objecter ms_handle_connect 0x7f464413f710 2024-03-25T18:21:27.572 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.571+0000 7f462bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:27.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007685218 0 0) 0x7f46441317f0 con 0x7f46441430c0 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4630004eb0 con 0x7f46441430c0 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744220631 0 0) 0x7f4644131660 con 0x7f464413f710 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46441317f0 con 0x7f464413f710 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 client.?.objecter ms_handle_connect 0x7f46440a98f0 2024-03-25T18:21:27.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.578+0000 7f462bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:27.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.590+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841573890 0 0) 0x7f464413be70 con 0x7f46440a98f0 2024-03-25T18:21:27.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.590+0000 7f462bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.590+0000 7f462bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.590+0000 7f462bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.590+0000 7f462bfff640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:21:27.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.590+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4644131660 con 0x7f46440a98f0 2024-03-25T18:21:27.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.593+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 692319892 0 0) 0x7f4630004eb0 con 0x7f46441430c0 2024-03-25T18:21:27.594 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.593+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f464413be70 con 0x7f46441430c0 2024-03-25T18:21:27.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.595+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f463c001d30 con 0x7f46441430c0 2024-03-25T18:21:27.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.595+0000 7f462bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:27.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.595+0000 7f462bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:27.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.595+0000 7f462bfff640 10 monclient(hunting): dump: 2024-03-25T18:21:27.596 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:27.596 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:27.596 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.597 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.597 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:27.597 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:27.597 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:27.597 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:27.597 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:27.597 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:27.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.595+0000 7f462bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.600+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2537776629 0 0) 0x7f46441317f0 con 0x7f464413f710 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.600+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4630004eb0 con 0x7f464413f710 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.600+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f4638001bc0 con 0x7f464413f710 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.600+0000 7f462bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.600+0000 7f462bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.600+0000 7f462bfff640 10 monclient(hunting): dump: 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:27.602 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:27.603 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:27.603 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:27.603 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:27.603 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:27.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.604+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3738088351 0 0) 0x7f464413be70 con 0x7f46441430c0 2024-03-25T18:21:27.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.604+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 >> v1:172.21.15.191:6789/0 conn(0x7f464413f710 legacy=0x7f46441d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.604+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 >> v1:172.21.15.162:6789/0 conn(0x7f46440a98f0 legacy=0x7f4644130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.604+0000 7f462bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:27.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.604+0000 7f462bfff640 1 monclient: found mon.c 2024-03-25T18:21:27.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.604+0000 7f462bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:27.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f462bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:27.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644131980 con 0x7f46441430c0 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f462bfff640 10 monclient: _finish_auth 0 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f462bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f462bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 10 monclient: _renew_subs 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4644135c60 con 0x7f46441430c0 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 10 monclient: _renew_subs 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46440b7770 con 0x7f46441430c0 2024-03-25T18:21:27.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.605+0000 7f464a16b640 1 librados: init done 2024-03-25T18:21:27.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.606+0000 7f464a16b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:27.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.606+0000 7f464a16b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:27.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.606+0000 7f464a16b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:27.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.606+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46441371d0 con 0x7f46441430c0 2024-03-25T18:21:27.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.607+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f463c003970 con 0x7f46441430c0 2024-03-25T18:21:27.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.607+0000 7f462bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:27.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.607+0000 7f4641ffb640 4 set_mon_vals no callback set 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.892+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f463c003d80 con 0x7f46441430c0 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.892+0000 7f462bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.892+0000 7f462bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.892+0000 7f462bfff640 10 monclient: dump: 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:27.894 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:27.895 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:27.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.893+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f463c003f60 con 0x7f46441430c0 2024-03-25T18:21:27.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.894+0000 7f46437fe640 10 monclient: get_auth_request con 0x7f463006dfa0 auth_method 0 2024-03-25T18:21:27.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.894+0000 7f462bfff640 10 client.4553.objecter ms_handle_connect 0x7f463006dfa0 2024-03-25T18:21:27.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.900+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f463c054b30 con 0x7f46441430c0 2024-03-25T18:21:27.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.900+0000 7f462bfff640 10 client.4553.objecter ms_dispatch 0x7f4644000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:27.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.900+0000 7f462bfff640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:27.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.900+0000 7f462bfff640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:27.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.900+0000 7f462bfff640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T18:21:27.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.900+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f463c050640 con 0x7f46441430c0 2024-03-25T18:21:27.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.901+0000 7f462bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:27.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:27.901+0000 7f462bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:28.131 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.129+0000 7f464a16b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:28.131 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.129+0000 7f464a16b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:28.131 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.129+0000 7f464a16b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:21:28.131 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.129+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4644074cc0 con 0x7f46441430c0 2024-03-25T18:21:28.161 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.160+0000 7f462bfff640 1 -- 172.21.15.162:0/3877522325 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f463c052570 con 0x7f46441430c0 2024-03-25T18:21:28.161 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.160+0000 7f462bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:28.161 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.160+0000 7f462bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:28.162 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:21:28.162 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:28.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 10 librados: watch_flush enter 2024-03-25T18:21:28.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 10 librados: watch_flush exit 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 10 client.4553.objecter successfully canceled tick 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 >> v1:172.21.15.191:6800/1552703795 conn(0x7f463006dfa0 legacy=0x7f4630070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 10 monclient: shutdown 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 >> v1:172.21.15.162:6790/0 conn(0x7f46441430c0 legacy=0x7f46441e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 shutdown_connections 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 >> 172.21.15.162:0/3877522325 conn(0x7f46440b7ea0 msgr2=0x7f4644074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 shutdown_connections 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.162+0000 7f464a16b640 1 -- 172.21.15.162:0/3877522325 wait complete. 2024-03-25T18:21:28.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:28.163+0000 7f464a16b640 1 librados: shutdown 2024-03-25T18:21:28.176 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:34.983 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:35.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.099+0000 7f799cbaf640 10 monclient: get_monmap_and_config 2024-03-25T18:21:35.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.100+0000 7f799cbaf640 10 monclient: build_initial_monmap 2024-03-25T18:21:35.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.100+0000 7f799cbaf640 10 monclient: monmap: 2024-03-25T18:21:35.101 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:35.101 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:35.101 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:21:35.101065+0000 2024-03-25T18:21:35.101 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:21:35.101065+0000 2024-03-25T18:21:35.101 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:35.102 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:35.102 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:35.102 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:21:35.102 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:21:35.102 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.100+0000 7f799cbaf640 1 Processor -- start 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.100+0000 7f799cbaf640 1 -- start start 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.100+0000 7f799cbaf640 10 monclient: init 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 10 monclient(hunting): picked mon.noname-b con 0x7f79980a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 10 monclient(hunting): picked mon.noname-c con 0x7f799813f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 10 monclient(hunting): picked mon.noname-a con 0x7f79981430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7998142520 con 0x7f79981430c0 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79980b28a0 con 0x7f799813f710 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79980b2a30 con 0x7f79980a98f0 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:35.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.101+0000 7f799cbaf640 10 monclient(hunting): authenticate will time out at 7994.375483s 2024-03-25T18:21:35.114 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.112+0000 7f7996d9d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f79981430c0 0x7f799814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55760/0 (socket says 172.21.15.162:55760) 2024-03-25T18:21:35.114 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.113+0000 7f7996d9d640 1 -- 172.21.15.162:0/1419425243 learned_addr learned my addr 172.21.15.162:0/1419425243 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:35.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.125+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438332897 0 0) 0x7f7998142520 con 0x7f79981430c0 2024-03-25T18:21:35.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.125+0000 7f7995d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.125+0000 7f7995d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.125+0000 7f7995d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.125+0000 7f7995d9b640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:21:35.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.125+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7984003520 con 0x7f79981430c0 2024-03-25T18:21:35.136 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2197321229 0 0) 0x7f7984003520 con 0x7f79981430c0 2024-03-25T18:21:35.136 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 >> v1:172.21.15.162:6790/0 conn(0x7f79980a98f0 legacy=0x7f79980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 >> v1:172.21.15.191:6789/0 conn(0x7f799813f710 legacy=0x7f79981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 1 monclient: found mon.noname-a 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79980b44c0 con 0x7f79981430c0 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 10 monclient: _finish_auth 0 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7990001a10 con 0x7f79981430c0 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f799cbaf640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f799cbaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f7995d9b640 10 monclient: dump: 2024-03-25T18:21:35.137 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:35.138 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.135+0000 7f799cbaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7990001bf0 con 0x7f79981430c0 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f7995d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f7995d9b640 1 -- 172.21.15.162:0/1419425243 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7990002000 con 0x7f79981430c0 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f799cbaf640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f799cbaf640 4 set_mon_vals no callback set 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f7995d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f7995d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f7995d9b640 10 monclient: dump: 2024-03-25T18:21:35.147 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:35.148 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f799cbaf640 10 monclient: shutdown 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f799cbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f799cbaf640 1 -- 172.21.15.162:0/1419425243 >> v1:172.21.15.162:6789/0 conn(0x7f79981430c0 legacy=0x7f799814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f799cbaf640 1 -- 172.21.15.162:0/1419425243 shutdown_connections 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.146+0000 7f799cbaf640 1 -- 172.21.15.162:0/1419425243 >> 172.21.15.162:0/1419425243 conn(0x7f79980b7ea0 msgr2=0x7f79980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.147+0000 7f799cbaf640 1 -- 172.21.15.162:0/1419425243 shutdown_connections 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.147+0000 7f799cbaf640 1 -- 172.21.15.162:0/1419425243 wait complete. 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 10 monclient: build_initial_monmap 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 10 monclient: monmap: 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:35.149 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:21:35.150 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:21:35.150 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:35.150 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:21:35.150 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:35.150 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:21:35.150 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:21:35.150 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:35.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 1 librados: starting msgr at 2024-03-25T18:21:35.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 1 librados: starting objecter 2024-03-25T18:21:35.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 1 Processor -- start 2024-03-25T18:21:35.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 1 -- start start 2024-03-25T18:21:35.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 1 librados: setting wanted keys 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 1 librados: calling monclient init 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.148+0000 7f799cbaf640 10 monclient: init 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 10 monclient(hunting): picked mon.noname-c con 0x7f79980a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 10 monclient(hunting): picked mon.noname-a con 0x7f799813f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 10 monclient(hunting): picked mon.noname-b con 0x7f79981430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799814daf0 con 0x7f799813f710 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f799814e650 con 0x7f79981430c0 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79981319b0 con 0x7f79980a98f0 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:35.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.149+0000 7f799cbaf640 10 monclient(hunting): authenticate will time out at 7994.423305s 2024-03-25T18:21:35.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.156+0000 7f798ffff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f799813f710 0x7f7998130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:55792/0 (socket says 172.21.15.162:55792) 2024-03-25T18:21:35.157 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.156+0000 7f798ffff640 1 -- 172.21.15.162:0/528085871 learned_addr learned my addr 172.21.15.162:0/528085871 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:35.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.157+0000 7f798effd640 10 client.?.objecter ms_handle_connect 0x7f799813f710 2024-03-25T18:21:35.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.157+0000 7f798effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:35.164 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.163+0000 7f798effd640 10 client.?.objecter ms_handle_connect 0x7f79981430c0 2024-03-25T18:21:35.165 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.163+0000 7f798effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:35.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.167+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207888137 0 0) 0x7f799814daf0 con 0x7f799813f710 2024-03-25T18:21:35.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.167+0000 7f798effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.167+0000 7f798effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.167+0000 7f798effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.167+0000 7f798effd640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:21:35.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.167+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7978004eb0 con 0x7f799813f710 2024-03-25T18:21:35.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.173+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359415393 0 0) 0x7f799814e650 con 0x7f79981430c0 2024-03-25T18:21:35.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.173+0000 7f798effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.173+0000 7f798effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.173+0000 7f798effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.173+0000 7f798effd640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:21:35.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.173+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799814daf0 con 0x7f79981430c0 2024-03-25T18:21:35.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.174+0000 7f798effd640 10 client.?.objecter ms_handle_connect 0x7f79980a98f0 2024-03-25T18:21:35.175 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.174+0000 7f798effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.192+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2203662834 0 0) 0x7f7978004eb0 con 0x7f799813f710 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.192+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f799814e650 con 0x7f799813f710 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.193+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f79800019d0 con 0x7f799813f710 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.193+0000 7f798effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.193+0000 7f798effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.193+0000 7f798effd640 10 monclient(hunting): dump: 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:35.194 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.195 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.195 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.195 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:35.195 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:35.195 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:35.195 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:35.195 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:35.195 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.193+0000 7f798effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:35.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.199+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207820087 0 0) 0x7f79981319b0 con 0x7f79980a98f0 2024-03-25T18:21:35.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.199+0000 7f798effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:35.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.199+0000 7f798effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:35.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.199+0000 7f798effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:35.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.199+0000 7f798effd640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:21:35.200 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.199+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7978004eb0 con 0x7f79980a98f0 2024-03-25T18:21:35.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.201+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 779216087 0 0) 0x7f799814daf0 con 0x7f79981430c0 2024-03-25T18:21:35.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.201+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79981319b0 con 0x7f79981430c0 2024-03-25T18:21:35.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.201+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7990001f00 con 0x7f79981430c0 2024-03-25T18:21:35.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.201+0000 7f798effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.201+0000 7f798effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:35.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.201+0000 7f798effd640 10 monclient(hunting): dump: 2024-03-25T18:21:35.202 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:35.202 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:35.203 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.203 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.203 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.203 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:35.203 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:35.203 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:35.203 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:35.203 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:35.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1209389457 0 0) 0x7f799814e650 con 0x7f799813f710 2024-03-25T18:21:35.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 >> v1:172.21.15.162:6790/0 conn(0x7f79980a98f0 legacy=0x7f799814bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 >> v1:172.21.15.191:6789/0 conn(0x7f79981430c0 legacy=0x7f79981312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:35.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 1 monclient: found mon.a 2024-03-25T18:21:35.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799814dd20 con 0x7f799813f710 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 10 monclient: _finish_auth 0 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f798effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f799cbaf640 5 monclient: authenticate success, global_id 4575 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f799cbaf640 10 monclient: _renew_subs 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f799cbaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799814e000 con 0x7f799813f710 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f799cbaf640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f799cbaf640 10 monclient: _renew_subs 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f799cbaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.209+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f799814c4c0 con 0x7f799813f710 2024-03-25T18:21:35.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.210+0000 7f799cbaf640 1 librados: init done 2024-03-25T18:21:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.211+0000 7f799cbaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.211+0000 7f799cbaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.211+0000 7f799cbaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:35.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.211+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799814e420 con 0x7f799813f710 2024-03-25T18:21:35.522 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.521+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7980003610 con 0x7f799813f710 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.521+0000 7f798effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.521+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f7980003a20 con 0x7f799813f710 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.521+0000 7f798effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.521+0000 7f798effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.521+0000 7f798effd640 10 monclient: dump: 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:35.523 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:35.524 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:35.524 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:35.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.521+0000 7f799559a640 4 set_mon_vals no callback set 2024-03-25T18:21:35.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.522+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f7980003c40 con 0x7f799813f710 2024-03-25T18:21:35.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.523+0000 7f799659c640 10 monclient: get_auth_request con 0x7f797806dfb0 auth_method 0 2024-03-25T18:21:35.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.523+0000 7f798effd640 10 client.4575.objecter ms_handle_connect 0x7f797806dfb0 2024-03-25T18:21:35.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.531+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f79800547b0 con 0x7f799813f710 2024-03-25T18:21:35.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.531+0000 7f798effd640 10 client.4575.objecter ms_dispatch 0x7f7998000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:35.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.531+0000 7f798effd640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:35.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.531+0000 7f798effd640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:35.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.531+0000 7f798effd640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T18:21:35.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.531+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79800502c0 con 0x7f799813f710 2024-03-25T18:21:35.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.531+0000 7f798effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:35.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.531+0000 7f798effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:35.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.782+0000 7f799cbaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:35.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.782+0000 7f799cbaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:35.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.782+0000 7f799cbaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:35.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.782+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f799814e800 con 0x7f799813f710 2024-03-25T18:21:35.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.817+0000 7f798effd640 1 -- 172.21.15.162:0/528085871 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f79800521f0 con 0x7f799813f710 2024-03-25T18:21:35.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.817+0000 7f798effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:35.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.817+0000 7f798effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:35.819 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:21:35.819 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 10 librados: watch_flush enter 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 10 librados: watch_flush exit 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 10 client.4575.objecter successfully canceled tick 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 >> v1:172.21.15.191:6800/1552703795 conn(0x7f797806dfb0 legacy=0x7f7978070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 10 monclient: shutdown 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 >> v1:172.21.15.162:6789/0 conn(0x7f799813f710 legacy=0x7f7998130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 shutdown_connections 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 >> 172.21.15.162:0/528085871 conn(0x7f79980b7ea0 msgr2=0x7f799814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:35.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 shutdown_connections 2024-03-25T18:21:35.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 1 -- 172.21.15.162:0/528085871 wait complete. 2024-03-25T18:21:35.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:35.820+0000 7f799cbaf640 1 librados: shutdown 2024-03-25T18:21:35.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:38.263 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:50.491 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:38.273 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:50.501 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:38.317 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:07:50.544 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:38.333 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:50.549 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:38.341 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:50.557 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:38.341 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:50.569 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:38.347 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:50.574 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:38.370 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:07:50.586 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:38.375 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:07:50.603 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:42.641 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.744+0000 7fdb28d38640 10 monclient: get_monmap_and_config 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.744+0000 7fdb28d38640 10 monclient: build_initial_monmap 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.744+0000 7fdb28d38640 10 monclient: monmap: 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:21:42.745257+0000 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:21:42.745257+0000 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:42.745 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:42.746 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:42.746 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:21:42.746 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:21:42.746 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:42.747 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.744+0000 7fdb28d38640 1 Processor -- start 2024-03-25T18:21:42.747 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.744+0000 7fdb28d38640 1 -- start start 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.744+0000 7fdb28d38640 10 monclient: init 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 10 monclient(hunting): picked mon.noname-b con 0x7fdb2414be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240af940 con 0x7fdb240b1730 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240b0a70 con 0x7fdb240b4d60 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240b0d50 con 0x7fdb2414be90 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.745+0000 7fdb28d38640 10 monclient(hunting): authenticate will time out at 8002.019293s 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.746+0000 7fdb22d9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fdb2414be90 0x7fdb2414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:57248/0 (socket says 172.21.15.162:57248) 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.746+0000 7fdb22d9d640 1 -- 172.21.15.162:0/1529533823 learned_addr learned my addr 172.21.15.162:0/1529533823 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:42.748 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.746+0000 7fdb2259c640 1 --1- 172.21.15.162:0/1529533823 >> v1:172.21.15.191:6789/0 conn(0x7fdb240b4d60 0x7fdb2413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:36020/0 (socket says 172.21.15.162:36020) 2024-03-25T18:21:42.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.793+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564659197 0 0) 0x7fdb240af940 con 0x7fdb240b1730 2024-03-25T18:21:42.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.793+0000 7fdb2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:42.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.793+0000 7fdb2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:42.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.793+0000 7fdb2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:42.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.793+0000 7fdb2159a640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:21:42.795 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.793+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb0c003520 con 0x7fdb240b1730 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.805+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 715657446 0 0) 0x7fdb0c003520 con 0x7fdb240b1730 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.805+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 >> v1:172.21.15.162:6790/0 conn(0x7fdb2414be90 legacy=0x7fdb2414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.805+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 >> v1:172.21.15.191:6789/0 conn(0x7fdb240b4d60 legacy=0x7fdb2413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.805+0000 7fdb2159a640 10 monclient: _finish_hunting 0 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.805+0000 7fdb2159a640 1 monclient: found mon.noname-a 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.805+0000 7fdb2159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.805+0000 7fdb2159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.805+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240b1030 con 0x7fdb240b1730 2024-03-25T18:21:42.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb2159a640 10 monclient: _finish_auth 0 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb2159a640 10 monclient: _check_auth_tickets 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdb18001690 con 0x7fdb240b1730 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb2159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb2159a640 10 monclient: dump: 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:42.808 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:42.809 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:42.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb28d38640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:21:42.809 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.806+0000 7fdb28d38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:42.812 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.811+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb180017f0 con 0x7fdb240b1730 2024-03-25T18:21:42.812 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.811+0000 7fdb2159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.811+0000 7fdb2159a640 1 -- 172.21.15.162:0/1529533823 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdb18001c00 con 0x7fdb240b1730 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.811+0000 7fdb2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.811+0000 7fdb2159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.811+0000 7fdb2159a640 10 monclient: dump: 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:42.813 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:42.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.811+0000 7fdb28d38640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:42.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.811+0000 7fdb28d38640 4 set_mon_vals no callback set 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.812+0000 7fdb28d38640 10 monclient: shutdown 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.812+0000 7fdb28d38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.812+0000 7fdb28d38640 1 -- 172.21.15.162:0/1529533823 >> v1:172.21.15.162:6789/0 conn(0x7fdb240b1730 legacy=0x7fdb240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.812+0000 7fdb28d38640 1 -- 172.21.15.162:0/1529533823 shutdown_connections 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.812+0000 7fdb28d38640 1 -- 172.21.15.162:0/1529533823 >> 172.21.15.162:0/1529533823 conn(0x7fdb240617a0 msgr2=0x7fdb24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.812+0000 7fdb28d38640 1 -- 172.21.15.162:0/1529533823 shutdown_connections 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.812+0000 7fdb28d38640 1 -- 172.21.15.162:0/1529533823 wait complete. 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 10 monclient: build_initial_monmap 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 10 monclient: monmap: 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:42.816 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 1 librados: starting msgr at 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 1 librados: starting objecter 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 1 Processor -- start 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 1 -- start start 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 1 librados: setting wanted keys 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 1 librados: calling monclient init 2024-03-25T18:21:42.817 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.813+0000 7fdb28d38640 10 monclient: init 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 10 monclient(hunting): picked mon.noname-b con 0x7fdb2414be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb241d7940 con 0x7fdb240b4d60 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb241d8570 con 0x7fdb2414be90 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb241d8700 con 0x7fdb240b1730 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:42.818 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.814+0000 7fdb28d38640 10 monclient(hunting): authenticate will time out at 8002.087945s 2024-03-25T18:21:42.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.829+0000 7fdb21d9b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fdb240b4d60 0x7fdb24136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:60364/0 (socket says 172.21.15.162:60364) 2024-03-25T18:21:42.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.829+0000 7fdb21d9b640 1 -- 172.21.15.162:0/3489457522 learned_addr learned my addr 172.21.15.162:0/3489457522 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:42.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.831+0000 7fdb12ffd640 10 client.?.objecter ms_handle_connect 0x7fdb240b4d60 2024-03-25T18:21:42.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.831+0000 7fdb12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:42.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.844+0000 7fdb12ffd640 10 client.?.objecter ms_handle_connect 0x7fdb240b1730 2024-03-25T18:21:42.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.844+0000 7fdb12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:42.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.849+0000 7fdb12ffd640 10 client.?.objecter ms_handle_connect 0x7fdb2414be90 2024-03-25T18:21:42.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.849+0000 7fdb12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:42.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.854+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4207242499 0 0) 0x7fdb241d7940 con 0x7fdb240b4d60 2024-03-25T18:21:42.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.854+0000 7fdb12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:42.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.854+0000 7fdb12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:42.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.854+0000 7fdb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:42.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.854+0000 7fdb12ffd640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:21:42.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.855+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb04004eb0 con 0x7fdb240b4d60 2024-03-25T18:21:42.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.927+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3129530936 0 0) 0x7fdb04004eb0 con 0x7fdb240b4d60 2024-03-25T18:21:42.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.927+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb241d7940 con 0x7fdb240b4d60 2024-03-25T18:21:42.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.927+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdb18001690 con 0x7fdb240b4d60 2024-03-25T18:21:42.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.927+0000 7fdb12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.927+0000 7fdb12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.927+0000 7fdb12ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:42.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.927+0000 7fdb12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:42.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3040066171 0 0) 0x7fdb241d7940 con 0x7fdb240b4d60 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 >> v1:172.21.15.162:6790/0 conn(0x7fdb240b1730 legacy=0x7fdb241d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 >> v1:172.21.15.191:6789/0 conn(0x7fdb2414be90 legacy=0x7fdb2412fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 1 monclient: found mon.a 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb241361c0 con 0x7fdb240b4d60 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 10 monclient: _renew_subs 2024-03-25T18:21:42.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:42.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb24136470 con 0x7fdb240b4d60 2024-03-25T18:21:42.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:42.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 10 monclient: _renew_subs 2024-03-25T18:21:42.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:42.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb24137790 con 0x7fdb240b4d60 2024-03-25T18:21:42.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.935+0000 7fdb28d38640 1 librados: init done 2024-03-25T18:21:42.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.936+0000 7fdb28d38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:42.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.936+0000 7fdb28d38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:42.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.936+0000 7fdb28d38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:42.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:42.936+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb241d77a0 con 0x7fdb240b4d60 2024-03-25T18:21:43.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.288+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb180038a0 con 0x7fdb240b4d60 2024-03-25T18:21:43.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.288+0000 7fdb12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:43.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.288+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fdb18003b50 con 0x7fdb240b4d60 2024-03-25T18:21:43.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.288+0000 7fdb12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:43.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.288+0000 7fdb12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:43.290 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.288+0000 7fdb12ffd640 10 monclient: dump: 2024-03-25T18:21:43.290 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:43.290 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:43.291 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.288+0000 7fdb20d99640 4 set_mon_vals no callback set 2024-03-25T18:21:43.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.289+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fdb18003d70 con 0x7fdb240b4d60 2024-03-25T18:21:43.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.290+0000 7fdb2259c640 10 monclient: get_auth_request con 0x7fdb0406dc70 auth_method 0 2024-03-25T18:21:43.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.290+0000 7fdb12ffd640 10 client.4581.objecter ms_handle_connect 0x7fdb0406dc70 2024-03-25T18:21:43.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.296+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fdb18054840 con 0x7fdb240b4d60 2024-03-25T18:21:43.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.296+0000 7fdb12ffd640 10 client.4581.objecter ms_dispatch 0x7fdb24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:43.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.296+0000 7fdb12ffd640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:43.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.296+0000 7fdb12ffd640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:43.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.296+0000 7fdb12ffd640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T18:21:43.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.296+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb18050350 con 0x7fdb240b4d60 2024-03-25T18:21:43.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.296+0000 7fdb12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:43.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.296+0000 7fdb12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:43.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.550+0000 7fdb28d38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:43.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.550+0000 7fdb28d38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:43.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.551+0000 7fdb28d38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:43.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.551+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb240627b0 con 0x7fdb240b4d60 2024-03-25T18:21:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.587+0000 7fdb12ffd640 1 -- 172.21.15.162:0/3489457522 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fdb18052280 con 0x7fdb240b4d60 2024-03-25T18:21:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.587+0000 7fdb12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:43.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.587+0000 7fdb12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:43.588 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:21:43.588 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.589+0000 7fdb28d38640 10 librados: watch_flush enter 2024-03-25T18:21:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.589+0000 7fdb28d38640 10 librados: watch_flush exit 2024-03-25T18:21:43.591 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.589+0000 7fdb28d38640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.589+0000 7fdb28d38640 10 client.4581.objecter successfully canceled tick 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 >> v1:172.21.15.191:6800/1552703795 conn(0x7fdb0406dc70 legacy=0x7fdb04070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 10 monclient: shutdown 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 >> v1:172.21.15.162:6789/0 conn(0x7fdb240b4d60 legacy=0x7fdb24136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 shutdown_connections 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 >> 172.21.15.162:0/3489457522 conn(0x7fdb240617a0 msgr2=0x7fdb24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 shutdown_connections 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 1 -- 172.21.15.162:0/3489457522 wait complete. 2024-03-25T18:21:43.592 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:43.590+0000 7fdb28d38640 1 librados: shutdown 2024-03-25T18:21:43.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:50.410 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:50.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.527+0000 7f3d65a2a640 10 monclient: get_monmap_and_config 2024-03-25T18:21:50.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.527+0000 7f3d65a2a640 10 monclient: build_initial_monmap 2024-03-25T18:21:50.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.527+0000 7f3d65a2a640 10 monclient: monmap: 2024-03-25T18:21:50.528 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:50.528 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:50.529 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:21:50.528194+0000 2024-03-25T18:21:50.529 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:21:50.528194+0000 2024-03-25T18:21:50.529 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:50.529 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:50.529 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:50.529 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:21:50.529 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:21:50.529 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.527+0000 7f3d65a2a640 1 Processor -- start 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.527+0000 7f3d65a2a640 1 -- start start 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.527+0000 7f3d65a2a640 10 monclient: init 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f3d600a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6013f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f3d601430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d60142520 con 0x7f3d6013f710 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d600b28a0 con 0x7f3d601430c0 2024-03-25T18:21:50.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d600b2a30 con 0x7f3d600a98f0 2024-03-25T18:21:50.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:50.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.528+0000 7f3d65a2a640 10 monclient(hunting): authenticate will time out at 8009.802638s 2024-03-25T18:21:50.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.537+0000 7f3d5effd640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f3d600a98f0 0x7f3d600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:40006/0 (socket says 172.21.15.162:40006) 2024-03-25T18:21:50.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.537+0000 7f3d5effd640 1 -- 172.21.15.162:0/78824425 learned_addr learned my addr 172.21.15.162:0/78824425 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:50.557 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.556+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313989095 0 0) 0x7f3d600b2a30 con 0x7f3d600a98f0 2024-03-25T18:21:50.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.556+0000 7f3d5dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:50.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.556+0000 7f3d5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:50.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.556+0000 7f3d5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:50.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.556+0000 7f3d5dffb640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:21:50.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.556+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48003520 con 0x7f3d600a98f0 2024-03-25T18:21:50.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.566+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036384176 0 0) 0x7f3d600b28a0 con 0x7f3d601430c0 2024-03-25T18:21:50.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.566+0000 7f3d5dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:50.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.566+0000 7f3d5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:50.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.566+0000 7f3d5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:50.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.566+0000 7f3d5dffb640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:21:50.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.566+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d600b2a30 con 0x7f3d601430c0 2024-03-25T18:21:50.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.567+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3931677445 0 0) 0x7f3d48003520 con 0x7f3d600a98f0 2024-03-25T18:21:50.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.567+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 >> v1:172.21.15.191:6789/0 conn(0x7f3d601430c0 legacy=0x7f3d6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.567+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 >> v1:172.21.15.162:6789/0 conn(0x7f3d6013f710 legacy=0x7f3d601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.567+0000 7f3d5dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.567+0000 7f3d5dffb640 1 monclient: found mon.noname-b 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.567+0000 7f3d5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.567+0000 7f3d5dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d600b44c0 con 0x7f3d600a98f0 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d5dffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d5dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3d50001690 con 0x7f3d600a98f0 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d5dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d5dffb640 10 monclient: dump: 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:50.570 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:50.571 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:50.571 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:50.571 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:50.571 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:50.571 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:50.571 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:50.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d65a2a640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:21:50.571 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.568+0000 7f3d65a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.577+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d500017f0 con 0x7f3d600a98f0 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.577+0000 7f3d5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.577+0000 7f3d5dffb640 1 -- 172.21.15.162:0/78824425 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3d50001c00 con 0x7f3d600a98f0 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.577+0000 7f3d5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.577+0000 7f3d5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.577+0000 7f3d5dffb640 10 monclient: dump: 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:50.579 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:50.580 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:50.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.577+0000 7f3d65a2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:50.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.577+0000 7f3d65a2a640 4 set_mon_vals no callback set 2024-03-25T18:21:50.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.578+0000 7f3d65a2a640 10 monclient: shutdown 2024-03-25T18:21:50.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.578+0000 7f3d65a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:50.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.578+0000 7f3d65a2a640 1 -- 172.21.15.162:0/78824425 >> v1:172.21.15.162:6790/0 conn(0x7f3d600a98f0 legacy=0x7f3d600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.578+0000 7f3d65a2a640 1 -- 172.21.15.162:0/78824425 shutdown_connections 2024-03-25T18:21:50.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.578+0000 7f3d65a2a640 1 -- 172.21.15.162:0/78824425 >> 172.21.15.162:0/78824425 conn(0x7f3d600b7ea0 msgr2=0x7f3d600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:50.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.578+0000 7f3d65a2a640 1 -- 172.21.15.162:0/78824425 shutdown_connections 2024-03-25T18:21:50.581 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.578+0000 7f3d65a2a640 1 -- 172.21.15.162:0/78824425 wait complete. 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.579+0000 7f3d65a2a640 10 monclient: build_initial_monmap 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.579+0000 7f3d65a2a640 10 monclient: monmap: 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.579+0000 7f3d65a2a640 1 librados: starting msgr at 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.579+0000 7f3d65a2a640 1 librados: starting objecter 2024-03-25T18:21:50.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 1 Processor -- start 2024-03-25T18:21:50.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 1 -- start start 2024-03-25T18:21:50.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 1 librados: setting wanted keys 2024-03-25T18:21:50.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 1 librados: calling monclient init 2024-03-25T18:21:50.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 10 monclient: init 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f3d600a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.580+0000 7f3d65a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f3d601430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.581+0000 7f3d65a2a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d601db7d0 con 0x7f3d600a98f0 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.581+0000 7f3d65a2a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d600b1420 con 0x7f3d601430c0 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.581+0000 7f3d65a2a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d600b15b0 con 0x7f3d6013f710 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.581+0000 7f3d65a2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:50.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.581+0000 7f3d65a2a640 10 monclient(hunting): authenticate will time out at 8009.854778s 2024-03-25T18:21:50.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.584+0000 7f3d5f7fe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f3d601430c0 0x7f3d601d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:44372/0 (socket says 172.21.15.162:44372) 2024-03-25T18:21:50.585 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.584+0000 7f3d5f7fe640 1 -- 172.21.15.162:0/2633016919 learned_addr learned my addr 172.21.15.162:0/2633016919 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:50.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.585+0000 7f3d47fff640 10 client.?.objecter ms_handle_connect 0x7f3d601430c0 2024-03-25T18:21:50.587 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.585+0000 7f3d47fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:50.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.588+0000 7f3d47fff640 10 client.?.objecter ms_handle_connect 0x7f3d6013f710 2024-03-25T18:21:50.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.588+0000 7f3d47fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:50.641 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.641+0000 7f3d47fff640 10 client.?.objecter ms_handle_connect 0x7f3d600a98f0 2024-03-25T18:21:50.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.641+0000 7f3d47fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:50.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.657+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56040162 0 0) 0x7f3d601db7d0 con 0x7f3d600a98f0 2024-03-25T18:21:50.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.658+0000 7f3d47fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:50.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.658+0000 7f3d47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:50.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.658+0000 7f3d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:50.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.658+0000 7f3d47fff640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:21:50.659 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.658+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d4c004eb0 con 0x7f3d600a98f0 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.677+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1218368750 0 0) 0x7f3d4c004eb0 con 0x7f3d600a98f0 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.677+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d601db7d0 con 0x7f3d600a98f0 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.677+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3d50001bc0 con 0x7f3d600a98f0 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.677+0000 7f3d47fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:50.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.677+0000 7f3d47fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.677+0000 7f3d47fff640 10 monclient(hunting): dump: 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:50.679 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.677+0000 7f3d47fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:50.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2590478260 0 0) 0x7f3d601db7d0 con 0x7f3d600a98f0 2024-03-25T18:21:50.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 >> v1:172.21.15.162:6790/0 conn(0x7f3d6013f710 legacy=0x7f3d600b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 >> v1:172.21.15.191:6789/0 conn(0x7f3d601430c0 legacy=0x7f3d601d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 1 monclient: found mon.a 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d600b1740 con 0x7f3d600a98f0 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 5 monclient: authenticate success, global_id 4587 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 10 monclient: _renew_subs 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d601db1a0 con 0x7f3d600a98f0 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:50.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 10 monclient: _renew_subs 2024-03-25T18:21:50.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:50.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d600b08f0 con 0x7f3d600a98f0 2024-03-25T18:21:50.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.686+0000 7f3d65a2a640 1 librados: init done 2024-03-25T18:21:50.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.687+0000 7f3d65a2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:50.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.687+0000 7f3d65a2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:50.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.687+0000 7f3d65a2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:50.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:50.687+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d600627b0 con 0x7f3d600a98f0 2024-03-25T18:21:51.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.017+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d50004a90 con 0x7f3d600a98f0 2024-03-25T18:21:51.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.017+0000 7f3d47fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:51.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.017+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f3d50001250 con 0x7f3d600a98f0 2024-03-25T18:21:51.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.017+0000 7f3d47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:51.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.017+0000 7f3d47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:51.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.017+0000 7f3d47fff640 10 monclient: dump: 2024-03-25T18:21:51.019 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:51.020 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.017+0000 7f3d5d7fa640 4 set_mon_vals no callback set 2024-03-25T18:21:51.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.018+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f3d5001ac90 con 0x7f3d600a98f0 2024-03-25T18:21:51.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.019+0000 7f3d5e7fc640 10 monclient: get_auth_request con 0x7f3d4c06dc70 auth_method 0 2024-03-25T18:21:51.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.019+0000 7f3d47fff640 10 client.4587.objecter ms_handle_connect 0x7f3d4c06dc70 2024-03-25T18:21:51.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.024+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f3d50054950 con 0x7f3d600a98f0 2024-03-25T18:21:51.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.024+0000 7f3d47fff640 10 client.4587.objecter ms_dispatch 0x7f3d60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:51.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.024+0000 7f3d47fff640 3 client.4587.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:51.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.024+0000 7f3d47fff640 3 client.4587.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:51.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.024+0000 7f3d47fff640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-25T18:21:51.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.024+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d50050290 con 0x7f3d600a98f0 2024-03-25T18:21:51.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.024+0000 7f3d47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:51.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.024+0000 7f3d47fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:51.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.270+0000 7f3d65a2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:51.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.270+0000 7f3d65a2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:51.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.270+0000 7f3d65a2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:51.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.270+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d600abb10 con 0x7f3d600a98f0 2024-03-25T18:21:51.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.309+0000 7f3d47fff640 1 -- 172.21.15.162:0/2633016919 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f3d50052390 con 0x7f3d600a98f0 2024-03-25T18:21:51.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.309+0000 7f3d47fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:51.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.309+0000 7f3d47fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:51.311 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:21:51.311 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:51.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 10 librados: watch_flush enter 2024-03-25T18:21:51.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 10 librados: watch_flush exit 2024-03-25T18:21:51.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-25T18:21:51.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 10 client.4587.objecter successfully canceled tick 2024-03-25T18:21:51.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 >> v1:172.21.15.191:6800/1552703795 conn(0x7f3d4c06dc70 legacy=0x7f3d4c070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:51.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 10 monclient: shutdown 2024-03-25T18:21:51.314 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:51.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 >> v1:172.21.15.162:6789/0 conn(0x7f3d600a98f0 legacy=0x7f3d601daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:51.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 shutdown_connections 2024-03-25T18:21:51.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 >> 172.21.15.162:0/2633016919 conn(0x7f3d600b7ea0 msgr2=0x7f3d6013c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:51.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 shutdown_connections 2024-03-25T18:21:51.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 1 -- 172.21.15.162:0/2633016919 wait complete. 2024-03-25T18:21:51.315 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:51.312+0000 7f3d65a2a640 1 librados: shutdown 2024-03-25T18:21:51.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:58.131 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:58.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.248+0000 7fecfdfd7640 10 monclient: get_monmap_and_config 2024-03-25T18:21:58.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.248+0000 7fecfdfd7640 10 monclient: build_initial_monmap 2024-03-25T18:21:58.250 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.249+0000 7fecfdfd7640 10 monclient: monmap: 2024-03-25T18:21:58.250 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:58.250 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:58.250 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:21:58.250022+0000 2024-03-25T18:21:58.250 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:21:58.250022+0000 2024-03-25T18:21:58.251 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:58.251 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:58.251 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:58.251 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:21:58.251 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:21:58.251 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:58.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.250+0000 7fecfdfd7640 1 Processor -- start 2024-03-25T18:21:58.251 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.250+0000 7fecfdfd7640 1 -- start start 2024-03-25T18:21:58.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.250+0000 7fecfdfd7640 10 monclient: init 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 10 monclient(hunting): picked mon.noname-b con 0x7fecf80a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 10 monclient(hunting): picked mon.noname-a con 0x7fecf813f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 10 monclient(hunting): picked mon.noname-c con 0x7fecf81430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf8142520 con 0x7fecf813f710 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf80b28a0 con 0x7fecf81430c0 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf80b2a30 con 0x7fecf80a98f0 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:58.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.251+0000 7fecfdfd7640 10 monclient(hunting): authenticate will time out at 8017.525001s 2024-03-25T18:21:58.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.259+0000 7fecf7fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fecf81430c0 0x7fecf814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:36572/0 (socket says 172.21.15.162:36572) 2024-03-25T18:21:58.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.259+0000 7fecf7fff640 1 -- 172.21.15.162:0/3304296809 learned_addr learned my addr 172.21.15.162:0/3304296809 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:58.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.267+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870176253 0 0) 0x7fecf80b28a0 con 0x7fecf81430c0 2024-03-25T18:21:58.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.267+0000 7fecf67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.267+0000 7fecf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.267+0000 7fecf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:58.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.267+0000 7fecf67fc640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:21:58.268 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.267+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece0003520 con 0x7fecf81430c0 2024-03-25T18:21:58.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 235978336 0 0) 0x7fece0003520 con 0x7fecf81430c0 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 >> v1:172.21.15.162:6790/0 conn(0x7fecf80a98f0 legacy=0x7fecf80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 >> v1:172.21.15.162:6789/0 conn(0x7fecf813f710 legacy=0x7fecf81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 1 monclient: found mon.noname-c 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf80b44c0 con 0x7fecf81430c0 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fecf0001a10 con 0x7fecf81430c0 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:58.276 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecf67fc640 10 monclient: dump: 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecfdfd7640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:21:58.277 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.274+0000 7fecfdfd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:58.279 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf7fff640 1 -- 172.21.15.162:0/3304296809 >> v1:172.21.15.191:6789/0 conn(0x7fecf81430c0 legacy=0x7fecf814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf7fff640 1 -- 172.21.15.162:0/3304296809 >> v1:172.21.15.191:6789/0 conn(0x7fecf81430c0 legacy=0x7fecf814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf7fff640 1 --1- 172.21.15.162:0/3304296809 >> v1:172.21.15.191:6789/0 conn(0x7fecf81430c0 0x7fecf814dc30 :-1 s=OPENED pgs=57860 cs=1 l=1).handle_message read tag failed 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf7fff640 1 --1- 172.21.15.162:0/3304296809 >> v1:172.21.15.191:6789/0 conn(0x7fecf81430c0 0x7fecf814dc30 :-1 s=OPENED pgs=57860 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.191:6789/0 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 >> v1:172.21.15.191:6789/0 conn(0x7fecf81430c0 legacy=0x7fecf814dc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 10 monclient(hunting): picked mon.c con 0x7fecf80a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 10 monclient(hunting): picked mon.a con 0x7fecf813f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 10 monclient(hunting): picked mon.b con 0x7fece000b090 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fece0003520 con 0x7fecf813f710 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fecf80b28a0 con 0x7fece000b090 2024-03-25T18:21:58.280 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fece0010e10 con 0x7fecf80a98f0 2024-03-25T18:21:58.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:58.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.278+0000 7fecf67fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.191:6789/0 2024-03-25T18:21:58.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.294+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911225109 0 0) 0x7fecf80b28a0 con 0x7fece000b090 2024-03-25T18:21:58.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.294+0000 7fecf67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.294+0000 7fecf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.294+0000 7fecf67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:21:58.295 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.294+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 --> v1:172.21.15.191:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fece000fd40 con 0x7fece000b090 2024-03-25T18:21:58.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.300+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736105345 0 0) 0x7fece0010e10 con 0x7fecf80a98f0 2024-03-25T18:21:58.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.300+0000 7fecf67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.300+0000 7fecf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.300+0000 7fecf67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:21:58.301 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.300+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 --> v1:172.21.15.162:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fecf80b28a0 con 0x7fecf80a98f0 2024-03-25T18:21:58.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1421245779 0 0) 0x7fece000fd40 con 0x7fece000b090 2024-03-25T18:21:58.304 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 >> v1:172.21.15.162:6790/0 conn(0x7fecf80a98f0 legacy=0x7fece0003af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 >> v1:172.21.15.162:6789/0 conn(0x7fecf813f710 legacy=0x7fece0009970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 1 monclient: found mon.b 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7fece0012130 con 0x7fece000b090 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fecf0004dc0 con 0x7fece000b090 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecf67fc640 10 monclient: dump: 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:58.305 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:58.306 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:58.306 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:58.306 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:58.306 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:58.306 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:58.306 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:58.306 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:58.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.303+0000 7fecfdfd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:58.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.307+0000 7fecf67fc640 1 -- 172.21.15.162:0/3304296809 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecf00020a0 con 0x7fece000b090 2024-03-25T18:21:58.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.307+0000 7fecf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:58.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.307+0000 7fecfdfd7640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:58.308 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.307+0000 7fecfdfd7640 4 set_mon_vals no callback set 2024-03-25T18:21:58.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 10 monclient: shutdown 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 1 -- 172.21.15.162:0/3304296809 >> v1:172.21.15.191:6789/0 conn(0x7fece000b090 legacy=0x7fece000d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 1 -- 172.21.15.162:0/3304296809 shutdown_connections 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 1 -- 172.21.15.162:0/3304296809 >> 172.21.15.162:0/3304296809 conn(0x7fecf80b7ea0 msgr2=0x7fecf80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 1 -- 172.21.15.162:0/3304296809 shutdown_connections 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 1 -- 172.21.15.162:0/3304296809 wait complete. 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 10 monclient: build_initial_monmap 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.308+0000 7fecfdfd7640 10 monclient: monmap: 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:21:58.310 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:21:58.311 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:21:58.311 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:21:58.311 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:58.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.309+0000 7fecfdfd7640 1 librados: starting msgr at 2024-03-25T18:21:58.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.309+0000 7fecfdfd7640 1 librados: starting objecter 2024-03-25T18:21:58.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.309+0000 7fecfdfd7640 1 Processor -- start 2024-03-25T18:21:58.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.309+0000 7fecfdfd7640 1 -- start start 2024-03-25T18:21:58.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.309+0000 7fecfdfd7640 1 librados: setting wanted keys 2024-03-25T18:21:58.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.309+0000 7fecfdfd7640 1 librados: calling monclient init 2024-03-25T18:21:58.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.309+0000 7fecfdfd7640 10 monclient: init 2024-03-25T18:21:58.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:58.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:58.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 10 monclient(hunting): picked mon.noname-a con 0x7fece000b090 addr v1:172.21.15.162:6789/0 2024-03-25T18:21:58.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 10 monclient(hunting): picked mon.noname-b con 0x7fecf80a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:21:58.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 10 monclient(hunting): picked mon.noname-c con 0x7fecf813f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:21:58.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf813e650 con 0x7fece000b090 2024-03-25T18:21:58.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf813dce0 con 0x7fecf80a98f0 2024-03-25T18:21:58.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf813de70 con 0x7fecf813f710 2024-03-25T18:21:58.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:58.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.310+0000 7fecfdfd7640 10 monclient(hunting): authenticate will time out at 8017.584090s 2024-03-25T18:21:58.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.316+0000 7fecf7fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fecf813f710 0x7fecf8138f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:53024/0 (socket says 172.21.15.162:53024) 2024-03-25T18:21:58.317 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.316+0000 7fecf7fff640 1 -- 172.21.15.162:0/4182487680 learned_addr learned my addr 172.21.15.162:0/4182487680 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:21:58.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.318+0000 7fecdffff640 10 client.?.objecter ms_handle_connect 0x7fecf813f710 2024-03-25T18:21:58.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.318+0000 7fecdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:58.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.321+0000 7fecdffff640 10 client.?.objecter ms_handle_connect 0x7fece000b090 2024-03-25T18:21:58.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.321+0000 7fecdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:58.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.327+0000 7fecdffff640 10 client.?.objecter ms_handle_connect 0x7fecf80a98f0 2024-03-25T18:21:58.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.327+0000 7fecdffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:58.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.332+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409747951 0 0) 0x7fecf813e650 con 0x7fece000b090 2024-03-25T18:21:58.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.332+0000 7fecdffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:58.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.332+0000 7fecdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:58.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.332+0000 7fecdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:58.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.332+0000 7fecdffff640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:21:58.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.332+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece4004eb0 con 0x7fece000b090 2024-03-25T18:21:58.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.397+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 406894495 0 0) 0x7fece4004eb0 con 0x7fece000b090 2024-03-25T18:21:58.398 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.397+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecf813e650 con 0x7fece000b090 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.397+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fece8001930 con 0x7fece000b090 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.397+0000 7fecdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.397+0000 7fecdffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.397+0000 7fecdffff640 10 monclient(hunting): dump: 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:58.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.397+0000 7fecdffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:58.400 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:10.626 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 110757633 0 0) 0x7fecf813e650 con 0x7fece000b090 2024-03-25T18:21:58.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 >> v1:172.21.15.162:6790/0 conn(0x7fecf813f710 legacy=0x7fecf8138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 >> v1:172.21.15.191:6789/0 conn(0x7fecf80a98f0 legacy=0x7fecf81d7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:58.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 1 monclient: found mon.a 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:58.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf80b0ff0 con 0x7fece000b090 2024-03-25T18:21:58.411 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:10.637 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 10 monclient: _finish_auth 0 2024-03-25T18:21:58.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:58.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:58.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:21:58.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 10 monclient: _renew_subs 2024-03-25T18:21:58.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:58.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf80b1240 con 0x7fece000b090 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 10 monclient: _renew_subs 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf81d7a10 con 0x7fece000b090 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.409+0000 7fecfdfd7640 1 librados: init done 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.410+0000 7fecfdfd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.410+0000 7fecfdfd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.410+0000 7fecfdfd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:58.413 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.410+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf80abb20 con 0x7fece000b090 2024-03-25T18:21:58.466 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:10.682 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.470 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:10.698 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.474 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:10.690 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.493 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:10.721 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.505 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:10.721 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.519 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:10.746 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:58.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.753+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fece80035f0 con 0x7fece000b090 2024-03-25T18:21:58.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.753+0000 7fecdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:58.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.753+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fece8003760 con 0x7fece000b090 2024-03-25T18:21:58.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.753+0000 7fecdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:58.754 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.753+0000 7fecdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.753+0000 7fecdffff640 10 monclient: dump: 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:21:58.755 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.753+0000 7fecf5ffb640 4 set_mon_vals no callback set 2024-03-25T18:21:58.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.754+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fece80197b0 con 0x7fece000b090 2024-03-25T18:21:58.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.754+0000 7fecf6ffd640 10 monclient: get_auth_request con 0x7fece406dc70 auth_method 0 2024-03-25T18:21:58.756 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.754+0000 7fecdffff640 10 client.4593.objecter ms_handle_connect 0x7fece406dc70 2024-03-25T18:21:58.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.761+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fece8053450 con 0x7fece000b090 2024-03-25T18:21:58.762 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.761+0000 7fecdffff640 10 client.4593.objecter ms_dispatch 0x7fecf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:58.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.761+0000 7fecdffff640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:58.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.761+0000 7fecdffff640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:58.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.761+0000 7fecdffff640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T18:21:58.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.761+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fece804ed90 con 0x7fece000b090 2024-03-25T18:21:58.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.761+0000 7fecdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:58.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:58.761+0000 7fecdffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:58.765 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:10.993 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:59.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.005+0000 7fecfdfd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:59.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.005+0000 7fecfdfd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:59.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.005+0000 7fecfdfd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:21:59.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.005+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecf813db20 con 0x7fece000b090 2024-03-25T18:21:59.069 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.068+0000 7fecdffff640 1 -- 172.21.15.162:0/4182487680 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fece8050e90 con 0x7fece000b090 2024-03-25T18:21:59.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.068+0000 7fecdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:59.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.068+0000 7fecdffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:59.070 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:21:59.070 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 10 librados: watch_flush enter 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 10 librados: watch_flush exit 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 10 client.4593.objecter successfully canceled tick 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 >> v1:172.21.15.191:6800/1552703795 conn(0x7fece406dc70 legacy=0x7fece4070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 10 monclient: shutdown 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 >> v1:172.21.15.162:6789/0 conn(0x7fece000b090 legacy=0x7fecf813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 shutdown_connections 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 >> 172.21.15.162:0/4182487680 conn(0x7fecf80b7ea0 msgr2=0x7fecf80b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 shutdown_connections 2024-03-25T18:21:59.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 1 -- 172.21.15.162:0/4182487680 wait complete. 2024-03-25T18:21:59.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:21:59.071+0000 7fecfdfd7640 1 librados: shutdown 2024-03-25T18:21:59.085 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:05.892 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:06.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.010+0000 7f841470d640 10 monclient: get_monmap_and_config 2024-03-25T18:22:06.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.010+0000 7f841470d640 10 monclient: build_initial_monmap 2024-03-25T18:22:06.011 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.010+0000 7f841470d640 10 monclient: monmap: 2024-03-25T18:22:06.011 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:06.011 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:06.011 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:22:06.011257+0000 2024-03-25T18:22:06.011 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:22:06.011257+0000 2024-03-25T18:22:06.012 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:06.012 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:06.012 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:06.012 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:22:06.012 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:22:06.012 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:06.012 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.010+0000 7f841470d640 1 Processor -- start 2024-03-25T18:22:06.012 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.011+0000 7f841470d640 1 -- start start 2024-03-25T18:22:06.013 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.011+0000 7f841470d640 10 monclient: init 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 10 monclient(hunting): picked mon.noname-c con 0x7f840c0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 10 monclient(hunting): picked mon.noname-a con 0x7f840c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 10 monclient(hunting): picked mon.noname-b con 0x7f840c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c142520 con 0x7f840c13f710 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c0b28a0 con 0x7f840c0a98f0 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c0b2a30 con 0x7f840c1430c0 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:06.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.012+0000 7f841470d640 10 monclient(hunting): authenticate will time out at 8025.286060s 2024-03-25T18:22:06.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.022+0000 7f8411ca8640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f840c13f710 0x7f840c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:40384/0 (socket says 172.21.15.162:40384) 2024-03-25T18:22:06.024 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.022+0000 7f8411ca8640 1 -- 172.21.15.162:0/3749899212 learned_addr learned my addr 172.21.15.162:0/3749899212 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:06.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.039+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138815995 0 0) 0x7f840c142520 con 0x7f840c13f710 2024-03-25T18:22:06.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.039+0000 7f84114a7640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.039+0000 7f84114a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.039+0000 7f84114a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.039+0000 7f84114a7640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:22:06.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.039+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f8003520 con 0x7f840c13f710 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331114016 0 0) 0x7f840c0b28a0 con 0x7f840c0a98f0 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c142520 con 0x7f840c0a98f0 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732143271 0 0) 0x7f840c0b2a30 con 0x7f840c1430c0 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:22:06.044 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.042+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c0b28a0 con 0x7f840c1430c0 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1107158520 0 0) 0x7f83f8003520 con 0x7f840c13f710 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 >> v1:172.21.15.162:6790/0 conn(0x7f840c1430c0 legacy=0x7f840c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 >> v1:172.21.15.191:6789/0 conn(0x7f840c0a98f0 legacy=0x7f840c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 10 monclient: _finish_hunting 0 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 1 monclient: found mon.noname-a 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:06.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c0b44c0 con 0x7f840c13f710 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 10 monclient: _finish_auth 0 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 10 monclient: _check_auth_tickets 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f83fc001690 con 0x7f840c13f710 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f84114a7640 10 monclient: dump: 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:06.057 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:06.058 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:06.058 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:06.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f841470d640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:22:06.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.055+0000 7f841470d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.069+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83fc0017f0 con 0x7f840c13f710 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.069+0000 7f84114a7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.069+0000 7f84114a7640 1 -- 172.21.15.162:0/3749899212 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f83fc001c00 con 0x7f840c13f710 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.069+0000 7f841470d640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.069+0000 7f841470d640 4 set_mon_vals no callback set 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.069+0000 7f84114a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.069+0000 7f84114a7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.069+0000 7f84114a7640 10 monclient: dump: 2024-03-25T18:22:06.070 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:06.071 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:06.072 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.070+0000 7f841470d640 10 monclient: shutdown 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.070+0000 7f841470d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.070+0000 7f841470d640 1 -- 172.21.15.162:0/3749899212 >> v1:172.21.15.162:6789/0 conn(0x7f840c13f710 legacy=0x7f840c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.070+0000 7f841470d640 1 -- 172.21.15.162:0/3749899212 shutdown_connections 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.070+0000 7f841470d640 1 -- 172.21.15.162:0/3749899212 >> 172.21.15.162:0/3749899212 conn(0x7f840c0b7ea0 msgr2=0x7f840c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.070+0000 7f841470d640 1 -- 172.21.15.162:0/3749899212 shutdown_connections 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.070+0000 7f841470d640 1 -- 172.21.15.162:0/3749899212 wait complete. 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.071+0000 7f841470d640 10 monclient: build_initial_monmap 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.071+0000 7f841470d640 10 monclient: monmap: 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:06.073 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.071+0000 7f841470d640 1 librados: starting msgr at 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.071+0000 7f841470d640 1 librados: starting objecter 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.071+0000 7f841470d640 1 Processor -- start 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 1 -- start start 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 1 librados: setting wanted keys 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 1 librados: calling monclient init 2024-03-25T18:22:06.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 10 monclient: init 2024-03-25T18:22:06.075 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:06.075 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:06.075 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 10 monclient(hunting): picked mon.noname-b con 0x7f840c0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:06.075 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 10 monclient(hunting): picked mon.noname-c con 0x7f840c13f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 10 monclient(hunting): picked mon.noname-a con 0x7f840c1430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.072+0000 7f841470d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c137f10 con 0x7f840c1430c0 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.073+0000 7f841470d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c1dc070 con 0x7f840c0a98f0 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.073+0000 7f841470d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840c137100 con 0x7f840c13f710 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.073+0000 7f841470d640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:06.076 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.073+0000 7f841470d640 10 monclient(hunting): authenticate will time out at 8025.346701s 2024-03-25T18:22:06.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.088+0000 7f8411ca8640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f840c13f710 0x7f840c1db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:57880/0 (socket says 172.21.15.162:57880) 2024-03-25T18:22:06.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.088+0000 7f8411ca8640 1 -- 172.21.15.162:0/2705270881 learned_addr learned my addr 172.21.15.162:0/2705270881 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:06.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.091+0000 7f8402ffd640 10 client.?.objecter ms_handle_connect 0x7f840c13f710 2024-03-25T18:22:06.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.091+0000 7f8402ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.093+0000 7f8402ffd640 10 client.?.objecter ms_handle_connect 0x7f840c0a98f0 2024-03-25T18:22:06.094 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.093+0000 7f8402ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:06.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.098+0000 7f8402ffd640 10 client.?.objecter ms_handle_connect 0x7f840c1430c0 2024-03-25T18:22:06.099 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.098+0000 7f8402ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:06.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.100+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829148735 0 0) 0x7f840c137100 con 0x7f840c13f710 2024-03-25T18:22:06.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.100+0000 7f8402ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.100+0000 7f8402ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.100+0000 7f8402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.100+0000 7f8402ffd640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:22:06.101 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.100+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f0004eb0 con 0x7f840c13f710 2024-03-25T18:22:06.102 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.101+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684609426 0 0) 0x7f840c1dc070 con 0x7f840c0a98f0 2024-03-25T18:22:06.102 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.101+0000 7f8402ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.102 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.101+0000 7f8402ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.102 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.101+0000 7f8402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.101+0000 7f8402ffd640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:22:06.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.101+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c137100 con 0x7f840c0a98f0 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611408049 0 0) 0x7f840c137f10 con 0x7f840c1430c0 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f840c1dc070 con 0x7f840c1430c0 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2373565308 0 0) 0x7f840c137100 con 0x7f840c0a98f0 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f840c137f10 con 0x7f840c0a98f0 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8408001970 con 0x7f840c0a98f0 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:06.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2719551211 0 0) 0x7f83f0004eb0 con 0x7f840c13f710 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f840c137100 con 0x7f840c13f710 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f83fc001bc0 con 0x7f840c13f710 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.109+0000 7f8402ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:06.113 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:06.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 627611105 0 0) 0x7f840c137f10 con 0x7f840c0a98f0 2024-03-25T18:22:06.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 >> v1:172.21.15.162:6790/0 conn(0x7f840c13f710 legacy=0x7f840c1db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 >> v1:172.21.15.162:6789/0 conn(0x7f840c1430c0 legacy=0x7f840c1db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 1 monclient: found mon.b 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c1373e0 con 0x7f840c0a98f0 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f8402ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 10 monclient: _renew_subs 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840c1358e0 con 0x7f840c0a98f0 2024-03-25T18:22:06.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:06.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 10 monclient: _renew_subs 2024-03-25T18:22:06.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:06.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f840c136150 con 0x7f840c0a98f0 2024-03-25T18:22:06.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.121+0000 7f841470d640 1 librados: init done 2024-03-25T18:22:06.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.122+0000 7f841470d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:06.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.122+0000 7f841470d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:06.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.122+0000 7f841470d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:06.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.122+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f840c0627b0 con 0x7f840c0a98f0 2024-03-25T18:22:06.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.383+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8408003610 con 0x7f840c0a98f0 2024-03-25T18:22:06.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.383+0000 7f8402ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:06.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.383+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8408003920 con 0x7f840c0a98f0 2024-03-25T18:22:06.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.383+0000 7f8402ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.383+0000 7f8402ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.383+0000 7f8402ffd640 10 monclient: dump: 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:06.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.383+0000 7f8410ca6640 4 set_mon_vals no callback set 2024-03-25T18:22:06.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.384+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f84080198f0 con 0x7f840c0a98f0 2024-03-25T18:22:06.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.384+0000 7f8411ca8640 10 monclient: get_auth_request con 0x7f83f006df70 auth_method 0 2024-03-25T18:22:06.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.385+0000 7f8402ffd640 10 client.4579.objecter ms_handle_connect 0x7f83f006df70 2024-03-25T18:22:06.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.389+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f8408053580 con 0x7f840c0a98f0 2024-03-25T18:22:06.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.389+0000 7f8402ffd640 10 client.4579.objecter ms_dispatch 0x7f840c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:06.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.389+0000 7f8402ffd640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:06.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.389+0000 7f8402ffd640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:06.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.389+0000 7f8402ffd640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T18:22:06.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.389+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f840804f090 con 0x7f840c0a98f0 2024-03-25T18:22:06.390 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.389+0000 7f8402ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:06.391 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.389+0000 7f8402ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:06.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.614+0000 7f841470d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:06.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.614+0000 7f841470d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:06.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.614+0000 7f841470d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:06.616 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.614+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f840c13e250 con 0x7f840c0a98f0 2024-03-25T18:22:06.706 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.704+0000 7f8402ffd640 1 -- 172.21.15.162:0/2705270881 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f8408050fc0 con 0x7f840c0a98f0 2024-03-25T18:22:06.706 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.704+0000 7f8402ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:06.706 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.704+0000 7f8402ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:06.706 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:22:06.706 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:06.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 10 librados: watch_flush enter 2024-03-25T18:22:06.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 10 librados: watch_flush exit 2024-03-25T18:22:06.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T18:22:06.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 10 client.4579.objecter successfully canceled tick 2024-03-25T18:22:06.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 >> v1:172.21.15.191:6800/1552703795 conn(0x7f83f006df70 legacy=0x7f83f0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 10 monclient: shutdown 2024-03-25T18:22:06.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:06.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 >> v1:172.21.15.191:6789/0 conn(0x7f840c0a98f0 legacy=0x7f840c137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:06.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 shutdown_connections 2024-03-25T18:22:06.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 >> 172.21.15.162:0/2705270881 conn(0x7f840c0b7ea0 msgr2=0x7f840c14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:06.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 shutdown_connections 2024-03-25T18:22:06.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 1 -- 172.21.15.162:0/2705270881 wait complete. 2024-03-25T18:22:06.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:06.707+0000 7f841470d640 1 librados: shutdown 2024-03-25T18:22:06.722 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:13.528 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.640+0000 7f2cbb7d0640 10 monclient: get_monmap_and_config 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.640+0000 7f2cbb7d0640 10 monclient: build_initial_monmap 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.640+0000 7f2cbb7d0640 10 monclient: monmap: 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:22:13.641996+0000 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:22:13.641996+0000 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:22:13.642 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:13.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.641+0000 7f2cbb7d0640 1 Processor -- start 2024-03-25T18:22:13.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 1 -- start start 2024-03-25T18:22:13.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 10 monclient: init 2024-03-25T18:22:13.643 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb40a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb41430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb4142520 con 0x7f2cb413f710 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40b28a0 con 0x7f2cb41430c0 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40b2a30 con 0x7f2cb40a98f0 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:13.644 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.642+0000 7f2cbb7d0640 10 monclient(hunting): authenticate will time out at 8032.916585s 2024-03-25T18:22:13.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.650+0000 7f2cb8d6b640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2cb413f710 0x7f2cb41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:48090/0 (socket says 172.21.15.162:48090) 2024-03-25T18:22:13.651 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.650+0000 7f2cb8d6b640 1 -- 172.21.15.162:0/172072693 learned_addr learned my addr 172.21.15.162:0/172072693 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:13.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.666+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320294172 0 0) 0x7f2cb40b2a30 con 0x7f2cb40a98f0 2024-03-25T18:22:13.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.666+0000 7f2cabfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.666+0000 7f2cabfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.666+0000 7f2cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.666+0000 7f2cabfff640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:22:13.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.666+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca0003520 con 0x7f2cb40a98f0 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.672+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932332830 0 0) 0x7f2cb4142520 con 0x7f2cb413f710 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.672+0000 7f2cabfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.672+0000 7f2cabfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.672+0000 7f2cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.672+0000 7f2cabfff640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:22:13.673 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.672+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb40b2a30 con 0x7f2cb413f710 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.674+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1748102614 0 0) 0x7f2cb40b28a0 con 0x7f2cb41430c0 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.674+0000 7f2cabfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.674+0000 7f2cabfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.675 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.674+0000 7f2cabfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.674+0000 7f2cabfff640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:22:13.676 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.674+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb4142520 con 0x7f2cb41430c0 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 46519032 0 0) 0x7f2ca0003520 con 0x7f2cb40a98f0 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 >> v1:172.21.15.191:6789/0 conn(0x7f2cb41430c0 legacy=0x7f2cb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 >> v1:172.21.15.162:6789/0 conn(0x7f2cb413f710 legacy=0x7f2cb41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 1 monclient: found mon.noname-b 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb40b44c0 con 0x7f2cb40a98f0 2024-03-25T18:22:13.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.687+0000 7f2cabfff640 10 monclient: _finish_auth 0 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cabfff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cbb7d0640 5 monclient: authenticate success, global_id 4583 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cbb7d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2cb0001690 con 0x7f2cb40a98f0 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cabfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cabfff640 10 monclient: dump: 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:13.690 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:13.691 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:13.691 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:13.691 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:13.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.688+0000 7f2cbb7d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.697+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2cb00017f0 con 0x7f2cb40a98f0 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.697+0000 7f2cabfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.697+0000 7f2cabfff640 1 -- 172.21.15.162:0/172072693 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2cb0001c00 con 0x7f2cb40a98f0 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.697+0000 7f2cbb7d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.697+0000 7f2cbb7d0640 4 set_mon_vals no callback set 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.697+0000 7f2cabfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.697+0000 7f2cabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:13.698 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.697+0000 7f2cabfff640 10 monclient: dump: 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:13.699 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:13.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.698+0000 7f2cbb7d0640 10 monclient: shutdown 2024-03-25T18:22:13.700 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.698+0000 7f2cbb7d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.698+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/172072693 >> v1:172.21.15.162:6790/0 conn(0x7f2cb40a98f0 legacy=0x7f2cb40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.698+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/172072693 shutdown_connections 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.698+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/172072693 >> 172.21.15.162:0/172072693 conn(0x7f2cb40b7ea0 msgr2=0x7f2cb40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.698+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/172072693 shutdown_connections 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.698+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/172072693 wait complete. 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 10 monclient: build_initial_monmap 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 10 monclient: monmap: 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:13.701 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 1 librados: starting msgr at 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 1 librados: starting objecter 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 1 Processor -- start 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 1 -- start start 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 1 librados: setting wanted keys 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 1 librados: calling monclient init 2024-03-25T18:22:13.702 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.699+0000 7f2cbb7d0640 10 monclient: init 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2cb40a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2cb413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2cb41430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb41db7d0 con 0x7f2cb41430c0 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40b1420 con 0x7f2cb40a98f0 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb40b15b0 con 0x7f2cb413f710 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:13.703 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.700+0000 7f2cbb7d0640 10 monclient(hunting): authenticate will time out at 8032.974065s 2024-03-25T18:22:13.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.711+0000 7f2cb9d6d640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f2cb41430c0 0x7f2cb41d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:48106/0 (socket says 172.21.15.162:48106) 2024-03-25T18:22:13.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.711+0000 7f2cb9d6d640 1 -- 172.21.15.162:0/1888983999 learned_addr learned my addr 172.21.15.162:0/1888983999 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:13.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.712+0000 7f2ca9ffb640 10 client.?.objecter ms_handle_connect 0x7f2cb413f710 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.712+0000 7f2ca9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.713+0000 7f2ca9ffb640 10 client.?.objecter ms_handle_connect 0x7f2cb41430c0 2024-03-25T18:22:13.714 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.713+0000 7f2ca9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:13.719 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.717+0000 7f2ca9ffb640 10 client.?.objecter ms_handle_connect 0x7f2cb40a98f0 2024-03-25T18:22:13.719 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.717+0000 7f2ca9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:13.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.719+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689830123 0 0) 0x7f2cb40b15b0 con 0x7f2cb413f710 2024-03-25T18:22:13.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.719+0000 7f2ca9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.719+0000 7f2ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.719+0000 7f2ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.719+0000 7f2ca9ffb640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:22:13.720 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.719+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c98004eb0 con 0x7f2cb413f710 2024-03-25T18:22:13.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.721+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742959567 0 0) 0x7f2cb41db7d0 con 0x7f2cb41430c0 2024-03-25T18:22:13.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.721+0000 7f2ca9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.721+0000 7f2ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.721+0000 7f2ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.721+0000 7f2ca9ffb640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:22:13.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.721+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb40b15b0 con 0x7f2cb41430c0 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.729+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2259061656 0 0) 0x7f2c98004eb0 con 0x7f2cb413f710 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.729+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cb41db7d0 con 0x7f2cb413f710 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.729+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2ca4001970 con 0x7f2cb413f710 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.729+0000 7f2ca9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.730+0000 7f2ca9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.730+0000 7f2ca9ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:13.731 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:13.732 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:13.732 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:13.732 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:13.732 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:13.732 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.730+0000 7f2ca9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:13.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.733+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080748981 0 0) 0x7f2cb40b1420 con 0x7f2cb40a98f0 2024-03-25T18:22:13.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.733+0000 7f2ca9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:13.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.733+0000 7f2ca9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:13.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.733+0000 7f2ca9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:13.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.733+0000 7f2ca9ffb640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:22:13.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.733+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c98004eb0 con 0x7f2cb40a98f0 2024-03-25T18:22:13.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 321222684 0 0) 0x7f2cb41db7d0 con 0x7f2cb413f710 2024-03-25T18:22:13.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 >> v1:172.21.15.191:6789/0 conn(0x7f2cb40a98f0 legacy=0x7f2cb41daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 >> v1:172.21.15.162:6789/0 conn(0x7f2cb41430c0 legacy=0x7f2cb41d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:13.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 1 monclient: found mon.c 2024-03-25T18:22:13.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:13.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb40b1740 con 0x7f2cb413f710 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2ca9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2cbb7d0640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2cbb7d0640 10 monclient: _renew_subs 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2cbb7d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.736+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb41db1a0 con 0x7f2cb413f710 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.737+0000 7f2cbb7d0640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.737+0000 7f2cbb7d0640 10 monclient: _renew_subs 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.737+0000 7f2cbb7d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.737+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb40b08f0 con 0x7f2cb413f710 2024-03-25T18:22:13.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.737+0000 7f2cbb7d0640 1 librados: init done 2024-03-25T18:22:13.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.738+0000 7f2cbb7d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:13.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.738+0000 7f2cbb7d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:13.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.738+0000 7f2cbb7d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:13.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:13.738+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb40627b0 con 0x7f2cb413f710 2024-03-25T18:22:14.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.034+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ca40047e0 con 0x7f2cb413f710 2024-03-25T18:22:14.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.034+0000 7f2ca9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:14.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.034+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2ca4004af0 con 0x7f2cb413f710 2024-03-25T18:22:14.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.034+0000 7f2ca9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:14.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.034+0000 7f2ca9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.034+0000 7f2ca9ffb640 10 monclient: dump: 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.034+0000 7f2cab7fe640 4 set_mon_vals no callback set 2024-03-25T18:22:14.037 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.035+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f2ca401aac0 con 0x7f2cb413f710 2024-03-25T18:22:14.038 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.036+0000 7f2cb956c640 10 monclient: get_auth_request con 0x7f2c9806de00 auth_method 0 2024-03-25T18:22:14.038 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.036+0000 7f2ca9ffb640 10 client.4586.objecter ms_handle_connect 0x7f2c9806de00 2024-03-25T18:22:14.042 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.041+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f2ca4054780 con 0x7f2cb413f710 2024-03-25T18:22:14.042 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.041+0000 7f2ca9ffb640 10 client.4586.objecter ms_dispatch 0x7f2cb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:14.042 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.041+0000 7f2ca9ffb640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:14.042 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.041+0000 7f2ca9ffb640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:14.042 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.041+0000 7f2ca9ffb640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T18:22:14.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.041+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ca4050290 con 0x7f2cb413f710 2024-03-25T18:22:14.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.041+0000 7f2ca9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:14.043 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.041+0000 7f2ca9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:14.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.291+0000 7f2cbb7d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:14.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.291+0000 7f2cbb7d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:14.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.291+0000 7f2cbb7d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:14.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.291+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cb40abb10 con 0x7f2cb413f710 2024-03-25T18:22:14.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.323+0000 7f2ca9ffb640 1 -- 172.21.15.162:0/1888983999 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f2ca40521c0 con 0x7f2cb413f710 2024-03-25T18:22:14.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.324+0000 7f2ca9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:14.325 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.324+0000 7f2ca9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:14.325 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:22:14.325 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 10 librados: watch_flush enter 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 10 librados: watch_flush exit 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 10 client.4586.objecter successfully canceled tick 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 >> v1:172.21.15.191:6800/1552703795 conn(0x7f2c9806de00 legacy=0x7f2c980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 10 monclient: shutdown 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 >> v1:172.21.15.162:6790/0 conn(0x7f2cb413f710 legacy=0x7f2cb40b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 shutdown_connections 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 >> 172.21.15.162:0/1888983999 conn(0x7f2cb40b7ea0 msgr2=0x7f2cb413c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 shutdown_connections 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 1 -- 172.21.15.162:0/1888983999 wait complete. 2024-03-25T18:22:14.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:14.326+0000 7f2cbb7d0640 1 librados: shutdown 2024-03-25T18:22:14.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:18.463 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:30.691 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.471 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:30.698 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.509 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:30.725 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.510 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:30.738 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.514 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:30.742 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.529 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:30.745 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.533 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:30.761 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.560 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:30.776 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.569 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:30.797 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:21.146 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.261+0000 7f133c115640 10 monclient: get_monmap_and_config 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.262+0000 7f133c115640 10 monclient: build_initial_monmap 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.262+0000 7f133c115640 10 monclient: monmap: 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:22:21.263068+0000 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:22:21.263068+0000 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:21.263 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:22:21.264 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:22:21.264 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.262+0000 7f133c115640 1 Processor -- start 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.262+0000 7f133c115640 1 -- start start 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.262+0000 7f133c115640 10 monclient: init 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 10 monclient(hunting): picked mon.noname-c con 0x7f13340a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 10 monclient(hunting): picked mon.noname-b con 0x7f133413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 10 monclient(hunting): picked mon.noname-a con 0x7f13341430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1334142520 con 0x7f13341430c0 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13340b28a0 con 0x7f13340a98f0 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13340b2a30 con 0x7f133413f710 2024-03-25T18:22:21.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:21.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.263+0000 7f133c115640 10 monclient(hunting): authenticate will time out at 8040.537444s 2024-03-25T18:22:21.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.268+0000 7f1339eb1640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f13340a98f0 0x7f13340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:58230/0 (socket says 172.21.15.162:58230) 2024-03-25T18:22:21.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.268+0000 7f1339eb1640 1 -- 172.21.15.162:0/2336873957 learned_addr learned my addr 172.21.15.162:0/2336873957 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.276+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003507407 0 0) 0x7f13340b28a0 con 0x7f13340a98f0 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.276+0000 7f1338eaf640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.276+0000 7f1338eaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.276+0000 7f1338eaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.276+0000 7f1338eaf640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:22:21.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.276+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1320003520 con 0x7f13340a98f0 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.280+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209790446 0 0) 0x7f1334142520 con 0x7f13341430c0 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.280+0000 7f1338eaf640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.280+0000 7f1338eaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.280+0000 7f1338eaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.280+0000 7f1338eaf640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:22:21.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.280+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13340b28a0 con 0x7f13341430c0 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 545160156 0 0) 0x7f1320003520 con 0x7f13340a98f0 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 >> v1:172.21.15.162:6790/0 conn(0x7f133413f710 legacy=0x7f13341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 >> v1:172.21.15.162:6789/0 conn(0x7f13341430c0 legacy=0x7f133414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 10 monclient: _finish_hunting 0 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 1 monclient: found mon.noname-c 2024-03-25T18:22:21.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13340b44c0 con 0x7f13340a98f0 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 10 monclient: _finish_auth 0 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 10 monclient: _check_auth_tickets 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1330001a10 con 0x7f13340a98f0 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f133c115640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f133c115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:21.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f1338eaf640 10 monclient: dump: 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:21.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.282+0000 7f133c115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:21.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.291+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1330001c30 con 0x7f13340a98f0 2024-03-25T18:22:21.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.291+0000 7f1338eaf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:21.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.291+0000 7f1338eaf640 1 -- 172.21.15.162:0/2336873957 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1330001fc0 con 0x7f13340a98f0 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.291+0000 7f133c115640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.291+0000 7f133c115640 4 set_mon_vals no callback set 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.291+0000 7f1338eaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.291+0000 7f1338eaf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.291+0000 7f1338eaf640 10 monclient: dump: 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:21.293 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.292+0000 7f133c115640 10 monclient: shutdown 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.292+0000 7f133c115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.292+0000 7f133c115640 1 -- 172.21.15.162:0/2336873957 >> v1:172.21.15.191:6789/0 conn(0x7f13340a98f0 legacy=0x7f13340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.292+0000 7f133c115640 1 -- 172.21.15.162:0/2336873957 shutdown_connections 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.292+0000 7f133c115640 1 -- 172.21.15.162:0/2336873957 >> 172.21.15.162:0/2336873957 conn(0x7f13340b7ea0 msgr2=0x7f13340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.292+0000 7f133c115640 1 -- 172.21.15.162:0/2336873957 shutdown_connections 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.292+0000 7f133c115640 1 -- 172.21.15.162:0/2336873957 wait complete. 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 10 monclient: build_initial_monmap 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 10 monclient: monmap: 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:21.296 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 1 librados: starting msgr at 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 1 librados: starting objecter 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 1 Processor -- start 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 1 -- start start 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 1 librados: setting wanted keys 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 1 librados: calling monclient init 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.293+0000 7f133c115640 10 monclient: init 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 10 monclient(hunting): picked mon.noname-b con 0x7f13340a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 10 monclient(hunting): picked mon.noname-c con 0x7f133413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 10 monclient(hunting): picked mon.noname-a con 0x7f13341430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:21.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1334139870 con 0x7f13341430c0 2024-03-25T18:22:21.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13341d51f0 con 0x7f13340a98f0 2024-03-25T18:22:21.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1334138830 con 0x7f133413f710 2024-03-25T18:22:21.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:21.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.294+0000 7f133c115640 10 monclient(hunting): authenticate will time out at 8040.568154s 2024-03-25T18:22:21.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.310+0000 7f1339eb1640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f13340a98f0 0x7f1334138120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:58280/0 (socket says 172.21.15.162:58280) 2024-03-25T18:22:21.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.310+0000 7f1339eb1640 1 -- 172.21.15.162:0/1973387084 learned_addr learned my addr 172.21.15.162:0/1973387084 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:21.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.312+0000 7f132a7fc640 10 client.?.objecter ms_handle_connect 0x7f13340a98f0 2024-03-25T18:22:21.313 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.312+0000 7f132a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:21.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.318+0000 7f132a7fc640 10 client.?.objecter ms_handle_connect 0x7f13341430c0 2024-03-25T18:22:21.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.318+0000 7f132a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:21.326 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.325+0000 7f132a7fc640 10 client.?.objecter ms_handle_connect 0x7f133413f710 2024-03-25T18:22:21.326 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.325+0000 7f132a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.334+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26748048 0 0) 0x7f1334138830 con 0x7f133413f710 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.334+0000 7f132a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.334+0000 7f132a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.334+0000 7f132a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.334+0000 7f132a7fc640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:22:21.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.334+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1318004eb0 con 0x7f133413f710 2024-03-25T18:22:21.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.336+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201777296 0 0) 0x7f13341d51f0 con 0x7f13340a98f0 2024-03-25T18:22:21.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.336+0000 7f132a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.336+0000 7f132a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.336+0000 7f132a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.336+0000 7f132a7fc640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:22:21.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.336+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1334138830 con 0x7f13340a98f0 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.352+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2304850662 0 0) 0x7f1334138830 con 0x7f13340a98f0 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.353+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13341d51f0 con 0x7f13340a98f0 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.353+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1330001950 con 0x7f13340a98f0 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.353+0000 7f132a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.353+0000 7f132a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.353+0000 7f132a7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:21.354 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:21.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.353+0000 7f132a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:21.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.359+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523822625 0 0) 0x7f1334139870 con 0x7f13341430c0 2024-03-25T18:22:21.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.359+0000 7f132a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:21.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.359+0000 7f132a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:21.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.359+0000 7f132a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:21.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.359+0000 7f132a7fc640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:22:21.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.359+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1334138830 con 0x7f13341430c0 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.361+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3180949634 0 0) 0x7f1318004eb0 con 0x7f133413f710 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.361+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1334139870 con 0x7f133413f710 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.361+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1324001930 con 0x7f133413f710 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.361+0000 7f132a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.361+0000 7f132a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.361+0000 7f132a7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:21.363 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:21.364 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:21.364 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:21.364 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:21.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1566431903 0 0) 0x7f13341d51f0 con 0x7f13340a98f0 2024-03-25T18:22:21.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 >> v1:172.21.15.162:6790/0 conn(0x7f133413f710 legacy=0x7f13341d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 >> v1:172.21.15.162:6789/0 conn(0x7f13341430c0 legacy=0x7f13341e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:21.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 1 monclient: found mon.b 2024-03-25T18:22:21.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:21.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:21.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13341d77a0 con 0x7f13340a98f0 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f132a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 5 monclient: authenticate success, global_id 4591 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 10 monclient: _renew_subs 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13341d83d0 con 0x7f13340a98f0 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 10 client.4591.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 10 monclient: _renew_subs 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1334138c80 con 0x7f13340a98f0 2024-03-25T18:22:21.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.377+0000 7f133c115640 1 librados: init done 2024-03-25T18:22:21.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.378+0000 7f133c115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:21.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.378+0000 7f133c115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:21.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.378+0000 7f133c115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:21.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.378+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13340b3e80 con 0x7f13340a98f0 2024-03-25T18:22:21.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.665+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1330001570 con 0x7f13340a98f0 2024-03-25T18:22:21.666 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.665+0000 7f132a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.665+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1330004fb0 con 0x7f13340a98f0 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.665+0000 7f132a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.665+0000 7f132a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.665+0000 7f132a7fc640 10 monclient: dump: 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:21.667 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.665+0000 7f132bfff640 4 set_mon_vals no callback set 2024-03-25T18:22:21.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.666+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f133001af80 con 0x7f13340a98f0 2024-03-25T18:22:21.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.667+0000 7f13396b0640 10 monclient: get_auth_request con 0x7f131806df70 auth_method 0 2024-03-25T18:22:21.668 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.667+0000 7f132a7fc640 10 client.4591.objecter ms_handle_connect 0x7f131806df70 2024-03-25T18:22:21.671 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.670+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f1330054c80 con 0x7f13340a98f0 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.670+0000 7f132a7fc640 10 client.4591.objecter ms_dispatch 0x7f1334000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.670+0000 7f132a7fc640 3 client.4591.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.670+0000 7f132a7fc640 3 client.4591.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.670+0000 7f132a7fc640 20 client.4591.objecter dump_active .. 0 homeless 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.670+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13300505c0 con 0x7f13340a98f0 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.670+0000 7f132a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:21.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.670+0000 7f132a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:21.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.895+0000 7f133c115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:21.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.895+0000 7f133c115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:21.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.895+0000 7f133c115640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:21.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.895+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13340627b0 con 0x7f13340a98f0 2024-03-25T18:22:21.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.938+0000 7f132a7fc640 1 -- 172.21.15.162:0/1973387084 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f13300526c0 con 0x7f13340a98f0 2024-03-25T18:22:21.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.938+0000 7f132a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:21.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.938+0000 7f132a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:21.939 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:22:21.939 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:21.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.940+0000 7f133c115640 10 librados: watch_flush enter 2024-03-25T18:22:21.942 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.940+0000 7f133c115640 10 librados: watch_flush exit 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.940+0000 7f133c115640 20 client.4591.objecter shutdown clearing up homeless session... 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.940+0000 7f133c115640 10 client.4591.objecter successfully canceled tick 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.940+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 >> v1:172.21.15.191:6800/1552703795 conn(0x7f131806df70 legacy=0x7f1318070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.940+0000 7f133c115640 10 monclient: shutdown 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.940+0000 7f133c115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.940+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 >> v1:172.21.15.191:6789/0 conn(0x7f13340a98f0 legacy=0x7f1334138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.941+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 shutdown_connections 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.941+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 >> 172.21.15.162:0/1973387084 conn(0x7f13340b7ea0 msgr2=0x7f1334074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.941+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 shutdown_connections 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.941+0000 7f133c115640 1 -- 172.21.15.162:0/1973387084 wait complete. 2024-03-25T18:22:21.943 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:21.941+0000 7f133c115640 1 librados: shutdown 2024-03-25T18:22:21.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:28.761 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:28.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.872+0000 7fd5030e6640 10 monclient: get_monmap_and_config 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.872+0000 7fd5030e6640 10 monclient: build_initial_monmap 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.872+0000 7fd5030e6640 10 monclient: monmap: 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:22:28.873663+0000 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:22:28.873663+0000 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:22:28.874 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:28.875 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.873+0000 7fd5030e6640 1 Processor -- start 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.873+0000 7fd5030e6640 1 -- start start 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.873+0000 7fd5030e6640 10 monclient: init 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd4fc0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd4fc13f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd4fc1430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc142520 con 0x7fd4fc1430c0 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0b28a0 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0b2a30 con 0x7fd4fc13f710 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:28.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.874+0000 7fd5030e6640 10 monclient(hunting): authenticate will time out at 8048.147862s 2024-03-25T18:22:28.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.878+0000 7fd500e82640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fd4fc0a98f0 0x7fd4fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:60936/0 (socket says 172.21.15.162:60936) 2024-03-25T18:22:28.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.878+0000 7fd500e82640 1 -- 172.21.15.162:0/2603919133 learned_addr learned my addr 172.21.15.162:0/2603919133 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:28.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.892+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803267481 0 0) 0x7fd4fc0b28a0 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.892+0000 7fd4fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.892+0000 7fd4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.892+0000 7fd4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.892+0000 7fd4fb7fe640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:22:28.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.892+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4ec003520 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.894+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151902193 0 0) 0x7fd4fc0b2a30 con 0x7fd4fc13f710 2024-03-25T18:22:28.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.894+0000 7fd4fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.894+0000 7fd4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.894+0000 7fd4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.894+0000 7fd4fb7fe640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:22:28.896 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.894+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4fc0b28a0 con 0x7fd4fc13f710 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3469342654 0 0) 0x7fd4ec003520 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 >> v1:172.21.15.162:6790/0 conn(0x7fd4fc13f710 legacy=0x7fd4fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 >> v1:172.21.15.162:6789/0 conn(0x7fd4fc1430c0 legacy=0x7fd4fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:22:28.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc0b44c0 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd4f0001690 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd4fb7fe640 10 monclient: dump: 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:28.901 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:28.902 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:28.902 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:28.902 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:28.902 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:28.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd5030e6640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:22:28.902 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.898+0000 7fd5030e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:28.906 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.905+0000 7fd4fb7fe640 1 -- 172.21.15.162:0/2603919133 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4f00017f0 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.905+0000 7fd4fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:28.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.905+0000 7fd5030e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:28.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.905+0000 7fd5030e6640 4 set_mon_vals no callback set 2024-03-25T18:22:28.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.906+0000 7fd5030e6640 10 monclient: shutdown 2024-03-25T18:22:28.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.906+0000 7fd5030e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:28.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.906+0000 7fd5030e6640 1 -- 172.21.15.162:0/2603919133 >> v1:172.21.15.191:6789/0 conn(0x7fd4fc0a98f0 legacy=0x7fd4fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.906+0000 7fd5030e6640 1 -- 172.21.15.162:0/2603919133 shutdown_connections 2024-03-25T18:22:28.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.906+0000 7fd5030e6640 1 -- 172.21.15.162:0/2603919133 >> 172.21.15.162:0/2603919133 conn(0x7fd4fc0b7ea0 msgr2=0x7fd4fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:28.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.906+0000 7fd5030e6640 1 -- 172.21.15.162:0/2603919133 shutdown_connections 2024-03-25T18:22:28.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.907+0000 7fd5030e6640 1 -- 172.21.15.162:0/2603919133 wait complete. 2024-03-25T18:22:28.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.907+0000 7fd5030e6640 10 monclient: build_initial_monmap 2024-03-25T18:22:28.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.907+0000 7fd5030e6640 10 monclient: monmap: 2024-03-25T18:22:28.909 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:28.909 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:28.909 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:22:28.909 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:22:28.909 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.908+0000 7fd5030e6640 1 librados: starting msgr at 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.908+0000 7fd5030e6640 1 librados: starting objecter 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.908+0000 7fd5030e6640 1 Processor -- start 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.908+0000 7fd5030e6640 1 -- start start 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.908+0000 7fd5030e6640 1 librados: setting wanted keys 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.908+0000 7fd5030e6640 1 librados: calling monclient init 2024-03-25T18:22:28.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.908+0000 7fd5030e6640 10 monclient: init 2024-03-25T18:22:28.911 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:28.911 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:28.911 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd4fc0a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:28.911 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd4fc13f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:28.911 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd4fc1430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:28.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0aba50 con 0x7fd4fc1430c0 2024-03-25T18:22:28.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0abd30 con 0x7fd4fc13f710 2024-03-25T18:22:28.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0ac010 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:28.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.909+0000 7fd5030e6640 10 monclient(hunting): authenticate will time out at 8048.182989s 2024-03-25T18:22:28.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.918+0000 7fd500e82640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fd4fc0a98f0 0x7fd4fc1363d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:33320/0 (socket says 172.21.15.162:33320) 2024-03-25T18:22:28.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.918+0000 7fd500e82640 1 -- 172.21.15.162:0/1863084594 learned_addr learned my addr 172.21.15.162:0/1863084594 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:28.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.919+0000 7fd4f97fa640 10 client.?.objecter ms_handle_connect 0x7fd4fc0a98f0 2024-03-25T18:22:28.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.919+0000 7fd4f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.923+0000 7fd4f97fa640 10 client.?.objecter ms_handle_connect 0x7fd4fc13f710 2024-03-25T18:22:28.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.923+0000 7fd4f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:28.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.925+0000 7fd4f97fa640 10 client.?.objecter ms_handle_connect 0x7fd4fc1430c0 2024-03-25T18:22:28.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.925+0000 7fd4f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.926+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221256159 0 0) 0x7fd4fc0abd30 con 0x7fd4fc13f710 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.926+0000 7fd4f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.926+0000 7fd4f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.926+0000 7fd4f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.926+0000 7fd4f97fa640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:22:28.928 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.926+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e0004eb0 con 0x7fd4fc13f710 2024-03-25T18:22:28.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.935+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1026088540 0 0) 0x7fd4e0004eb0 con 0x7fd4fc13f710 2024-03-25T18:22:28.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.935+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4fc0abd30 con 0x7fd4fc13f710 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.936+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd4e8001a30 con 0x7fd4fc13f710 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.936+0000 7fd4f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.936+0000 7fd4f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:28.937 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.936+0000 7fd4f97fa640 10 monclient(hunting): dump: 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:28.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.936+0000 7fd4f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.938+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946170321 0 0) 0x7fd4fc0ac010 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.938+0000 7fd4f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.938+0000 7fd4f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.938+0000 7fd4f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.938+0000 7fd4f97fa640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:22:28.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.938+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e0004eb0 con 0x7fd4fc0a98f0 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.940+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074524979 0 0) 0x7fd4fc0aba50 con 0x7fd4fc1430c0 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.940+0000 7fd4f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.940+0000 7fd4f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.940+0000 7fd4f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.940+0000 7fd4f97fa640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:22:28.941 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.940+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4fc0ac010 con 0x7fd4fc1430c0 2024-03-25T18:22:28.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3983063852 0 0) 0x7fd4fc0abd30 con 0x7fd4fc13f710 2024-03-25T18:22:28.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 >> v1:172.21.15.162:6790/0 conn(0x7fd4fc0a98f0 legacy=0x7fd4fc1363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 >> v1:172.21.15.162:6789/0 conn(0x7fd4fc1430c0 legacy=0x7fd4fc1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 1 monclient: found mon.b 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc138fe0 con 0x7fd4fc13f710 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd4f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 10 monclient: _renew_subs 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc137cf0 con 0x7fd4fc13f710 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 10 client.4597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 10 monclient: _renew_subs 2024-03-25T18:22:28.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc138ac0 con 0x7fd4fc13f710 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.954+0000 7fd5030e6640 1 librados: init done 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.955+0000 7fd5030e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.955+0000 7fd5030e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.955+0000 7fd5030e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:28.957 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:28.955+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc0627b0 con 0x7fd4fc13f710 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.233+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4e8001b90 con 0x7fd4fc13f710 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.233+0000 7fd4f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.233+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fd4e8003a60 con 0x7fd4fc13f710 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.233+0000 7fd4f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.233+0000 7fd4f97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.233+0000 7fd4f97fa640 10 monclient: dump: 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:29.235 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:29.236 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:29.236 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:29.236 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:29.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.233+0000 7fd4faffd640 4 set_mon_vals no callback set 2024-03-25T18:22:29.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.234+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fd4e8003c80 con 0x7fd4fc13f710 2024-03-25T18:22:29.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.235+0000 7fd500e82640 10 monclient: get_auth_request con 0x7fd4e006de00 auth_method 0 2024-03-25T18:22:29.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.235+0000 7fd4f97fa640 10 client.4597.objecter ms_handle_connect 0x7fd4e006de00 2024-03-25T18:22:29.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.242+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fd4e8053650 con 0x7fd4fc13f710 2024-03-25T18:22:29.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.242+0000 7fd4f97fa640 10 client.4597.objecter ms_dispatch 0x7fd4fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:29.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.242+0000 7fd4f97fa640 3 client.4597.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:29.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.242+0000 7fd4f97fa640 3 client.4597.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:29.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.242+0000 7fd4f97fa640 20 client.4597.objecter dump_active .. 0 homeless 2024-03-25T18:22:29.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.243+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4e804f160 con 0x7fd4fc13f710 2024-03-25T18:22:29.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.243+0000 7fd4f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:29.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.243+0000 7fd4f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:29.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.492+0000 7fd5030e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:29.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.492+0000 7fd5030e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:29.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.492+0000 7fd5030e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:29.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.492+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4fc139170 con 0x7fd4fc13f710 2024-03-25T18:22:29.527 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.526+0000 7fd4f97fa640 1 -- 172.21.15.162:0/1863084594 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fd4e8051090 con 0x7fd4fc13f710 2024-03-25T18:22:29.527 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.526+0000 7fd4f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:29.527 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.526+0000 7fd4f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:29.528 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:22:29.528 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:29.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.528+0000 7fd5030e6640 10 librados: watch_flush enter 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.528+0000 7fd5030e6640 10 librados: watch_flush exit 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.528+0000 7fd5030e6640 20 client.4597.objecter shutdown clearing up homeless session... 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.528+0000 7fd5030e6640 10 client.4597.objecter successfully canceled tick 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 >> v1:172.21.15.191:6800/1552703795 conn(0x7fd4e006de00 legacy=0x7fd4e00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 10 monclient: shutdown 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 >> v1:172.21.15.191:6789/0 conn(0x7fd4fc13f710 legacy=0x7fd4fc0ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 shutdown_connections 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 >> 172.21.15.162:0/1863084594 conn(0x7fd4fc0b7ea0 msgr2=0x7fd4fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 shutdown_connections 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 1 -- 172.21.15.162:0/1863084594 wait complete. 2024-03-25T18:22:29.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:29.529+0000 7fd5030e6640 1 librados: shutdown 2024-03-25T18:22:29.543 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:36.350 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.468+0000 7f8721182640 10 monclient: get_monmap_and_config 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.468+0000 7f8721182640 10 monclient: build_initial_monmap 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.468+0000 7f8721182640 10 monclient: monmap: 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:22:36.469143+0000 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:22:36.469143+0000 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:22:36.469 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:22:36.470 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.468+0000 7f8721182640 1 Processor -- start 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.468+0000 7f8721182640 1 -- start start 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.468+0000 7f8721182640 10 monclient: init 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 10 monclient(hunting): picked mon.noname-c con 0x7f871c0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 10 monclient(hunting): picked mon.noname-a con 0x7f871c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 10 monclient(hunting): picked mon.noname-b con 0x7f871c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c142520 con 0x7f871c13f710 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c0b28a0 con 0x7f871c0a98f0 2024-03-25T18:22:36.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c0b2a30 con 0x7f871c1430c0 2024-03-25T18:22:36.472 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:36.472 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.469+0000 7f8721182640 10 monclient(hunting): authenticate will time out at 8055.743234s 2024-03-25T18:22:36.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.475+0000 7f871259c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f871c13f710 0x7f871c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:35638/0 (socket says 172.21.15.162:35638) 2024-03-25T18:22:36.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.475+0000 7f871259c640 1 -- 172.21.15.162:0/1208575333 learned_addr learned my addr 172.21.15.162:0/1208575333 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:36.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.488+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724550123 0 0) 0x7f871c0b28a0 con 0x7f871c0a98f0 2024-03-25T18:22:36.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.488+0000 7f871a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:22:36.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.488+0000 7f871a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:36.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.488+0000 7f871a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:36.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.488+0000 7f871a59c640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:22:36.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.488+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8708003520 con 0x7f871c0a98f0 2024-03-25T18:22:36.496 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1677063539 0 0) 0x7f8708003520 con 0x7f871c0a98f0 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 >> v1:172.21.15.162:6790/0 conn(0x7f871c1430c0 legacy=0x7f871c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 >> v1:172.21.15.162:6789/0 conn(0x7f871c13f710 legacy=0x7f871c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 10 monclient: _finish_hunting 0 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 1 monclient: found mon.noname-c 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c0b44c0 con 0x7f871c0a98f0 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 10 monclient: _finish_auth 0 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 10 monclient: _check_auth_tickets 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f870c001690 con 0x7f871c0a98f0 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f871a59c640 10 monclient: dump: 2024-03-25T18:22:36.497 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f8721182640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:22:36.498 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.495+0000 7f8721182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.511+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f870c0017f0 con 0x7f871c0a98f0 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.511+0000 7f871a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.511+0000 7f871a59c640 1 -- 172.21.15.162:0/1208575333 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f870c001c40 con 0x7f871c0a98f0 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.511+0000 7f8721182640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.511+0000 7f8721182640 4 set_mon_vals no callback set 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.511+0000 7f871a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.511+0000 7f871a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.511+0000 7f871a59c640 10 monclient: dump: 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.513 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:36.514 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.512+0000 7f8721182640 10 monclient: shutdown 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.512+0000 7f8721182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.512+0000 7f8721182640 1 -- 172.21.15.162:0/1208575333 >> v1:172.21.15.191:6789/0 conn(0x7f871c0a98f0 legacy=0x7f871c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.512+0000 7f8721182640 1 -- 172.21.15.162:0/1208575333 shutdown_connections 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.512+0000 7f8721182640 1 -- 172.21.15.162:0/1208575333 >> 172.21.15.162:0/1208575333 conn(0x7f871c0b7ea0 msgr2=0x7f871c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:36.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.512+0000 7f8721182640 1 -- 172.21.15.162:0/1208575333 shutdown_connections 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.512+0000 7f8721182640 1 -- 172.21.15.162:0/1208575333 wait complete. 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.513+0000 7f8721182640 10 monclient: build_initial_monmap 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.513+0000 7f8721182640 10 monclient: monmap: 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.514+0000 7f8721182640 1 librados: starting msgr at 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.514+0000 7f8721182640 1 librados: starting objecter 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.514+0000 7f8721182640 1 Processor -- start 2024-03-25T18:22:36.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.514+0000 7f8721182640 1 -- start start 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.514+0000 7f8721182640 1 librados: setting wanted keys 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.514+0000 7f8721182640 1 librados: calling monclient init 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.514+0000 7f8721182640 10 monclient: init 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:36.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 10 monclient(hunting): picked mon.noname-b con 0x7f871c0a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 10 monclient(hunting): picked mon.noname-a con 0x7f871c13f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 10 monclient(hunting): picked mon.noname-c con 0x7f871c1430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c1df080 con 0x7f871c13f710 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c070420 con 0x7f871c0a98f0 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871c06f3c0 con 0x7f871c1430c0 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:36.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.515+0000 7f8721182640 10 monclient(hunting): authenticate will time out at 8055.788895s 2024-03-25T18:22:36.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.525+0000 7f871b59e640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f871c1430c0 0x7f871c1e64d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:38456/0 (socket says 172.21.15.162:38456) 2024-03-25T18:22:36.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.525+0000 7f871b59e640 1 -- 172.21.15.162:0/761932870 learned_addr learned my addr 172.21.15.162:0/761932870 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:36.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.527+0000 7f8713fff640 10 client.?.objecter ms_handle_connect 0x7f871c1430c0 2024-03-25T18:22:36.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.527+0000 7f8713fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:36.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.537+0000 7f8713fff640 10 client.?.objecter ms_handle_connect 0x7f871c0a98f0 2024-03-25T18:22:36.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.537+0000 7f8713fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:36.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.541+0000 7f8713fff640 10 client.?.objecter ms_handle_connect 0x7f871c13f710 2024-03-25T18:22:36.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.541+0000 7f8713fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:36.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.551+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406860505 0 0) 0x7f871c06f3c0 con 0x7f871c1430c0 2024-03-25T18:22:36.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.551+0000 7f8713fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:36.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.551+0000 7f8713fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:36.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.551+0000 7f8713fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:36.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.551+0000 7f8713fff640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:22:36.552 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.551+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86fc004eb0 con 0x7f871c1430c0 2024-03-25T18:22:36.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487736194 0 0) 0x7f871c070420 con 0x7f871c0a98f0 2024-03-25T18:22:36.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:36.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:36.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:36.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:22:36.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871c06f3c0 con 0x7f871c0a98f0 2024-03-25T18:22:36.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140847583 0 0) 0x7f871c1df080 con 0x7f871c13f710 2024-03-25T18:22:36.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:36.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:36.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:36.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:22:36.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.563+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f871c070420 con 0x7f871c13f710 2024-03-25T18:22:36.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.585+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 585390612 0 0) 0x7f86fc004eb0 con 0x7f871c1430c0 2024-03-25T18:22:36.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.585+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f871c1df080 con 0x7f871c1430c0 2024-03-25T18:22:36.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.585+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8714001c90 con 0x7f871c1430c0 2024-03-25T18:22:36.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.585+0000 7f8713fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.585+0000 7f8713fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:36.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.585+0000 7f8713fff640 10 monclient(hunting): dump: 2024-03-25T18:22:36.586 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:36.587 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.585+0000 7f8713fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:36.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.596+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2967986865 0 0) 0x7f871c06f3c0 con 0x7f871c0a98f0 2024-03-25T18:22:36.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.596+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86fc004eb0 con 0x7f871c0a98f0 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.596+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f870c001b40 con 0x7f871c0a98f0 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.596+0000 7f8713fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.596+0000 7f8713fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.596+0000 7f8713fff640 10 monclient(hunting): dump: 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:36.598 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:36.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.604+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1958059283 0 0) 0x7f871c070420 con 0x7f871c13f710 2024-03-25T18:22:36.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.604+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f871c06f3c0 con 0x7f871c13f710 2024-03-25T18:22:36.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.604+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8704001860 con 0x7f871c13f710 2024-03-25T18:22:36.605 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.604+0000 7f8713fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.604+0000 7f8713fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.604+0000 7f8713fff640 10 monclient(hunting): dump: 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:36.606 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:36.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.616+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3225770953 0 0) 0x7f871c1df080 con 0x7f871c1430c0 2024-03-25T18:22:36.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.616+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 >> v1:172.21.15.191:6789/0 conn(0x7f871c0a98f0 legacy=0x7f871c06ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.618 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 >> v1:172.21.15.162:6789/0 conn(0x7f871c13f710 legacy=0x7f871c1df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 1 monclient: found mon.c 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871c06f550 con 0x7f871c1430c0 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8713fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 10 monclient: _renew_subs 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c06e0a0 con 0x7f871c1430c0 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:36.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 10 monclient: _renew_subs 2024-03-25T18:22:36.620 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:36.620 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c1dfe00 con 0x7f871c1430c0 2024-03-25T18:22:36.620 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.617+0000 7f8721182640 1 librados: init done 2024-03-25T18:22:36.620 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.618+0000 7f8721182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:36.620 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.618+0000 7f8721182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:36.620 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.618+0000 7f8721182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:36.620 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.618+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871c1df230 con 0x7f871c1430c0 2024-03-25T18:22:36.938 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.937+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8714003990 con 0x7f871c1430c0 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.937+0000 7f8713fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.937+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8714003bc0 con 0x7f871c1430c0 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.937+0000 7f8713fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.937+0000 7f8713fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.937+0000 7f8713fff640 10 monclient: dump: 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:36.939 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:36.940 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:36.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.937+0000 7f8719d9b640 4 set_mon_vals no callback set 2024-03-25T18:22:36.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.937+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f8714003de0 con 0x7f871c1430c0 2024-03-25T18:22:36.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.938+0000 7f871ad9d640 10 monclient: get_auth_request con 0x7f86fc06e0f0 auth_method 0 2024-03-25T18:22:36.940 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.938+0000 7f8713fff640 10 client.4604.objecter ms_handle_connect 0x7f86fc06e0f0 2024-03-25T18:22:36.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.942+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f87140047e0 con 0x7f871c1430c0 2024-03-25T18:22:36.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.942+0000 7f8713fff640 10 client.4604.objecter ms_dispatch 0x7f871c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:36.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.942+0000 7f8713fff640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:36.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.942+0000 7f8713fff640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:36.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.942+0000 7f8713fff640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T18:22:36.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.942+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8714050460 con 0x7f871c1430c0 2024-03-25T18:22:36.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.942+0000 7f8713fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:36.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:36.942+0000 7f8713fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:37.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.166+0000 7f8721182640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:37.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.166+0000 7f8721182640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:37.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.166+0000 7f8721182640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:37.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.166+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f871c074cc0 con 0x7f871c1430c0 2024-03-25T18:22:37.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.216+0000 7f8713fff640 1 -- 172.21.15.162:0/761932870 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f8714052390 con 0x7f871c1430c0 2024-03-25T18:22:37.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.216+0000 7f8713fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:37.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.216+0000 7f8713fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:37.217 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:22:37.218 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 10 librados: watch_flush enter 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 10 librados: watch_flush exit 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 10 client.4604.objecter successfully canceled tick 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 >> v1:172.21.15.191:6800/1552703795 conn(0x7f86fc06e0f0 legacy=0x7f86fc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 10 monclient: shutdown 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 >> v1:172.21.15.162:6790/0 conn(0x7f871c1430c0 legacy=0x7f871c1e64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 shutdown_connections 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.218+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 >> 172.21.15.162:0/761932870 conn(0x7f871c0b7ea0 msgr2=0x7f871c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.219+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 shutdown_connections 2024-03-25T18:22:37.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.219+0000 7f8721182640 1 -- 172.21.15.162:0/761932870 wait complete. 2024-03-25T18:22:37.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:37.219+0000 7f8721182640 1 librados: shutdown 2024-03-25T18:22:37.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:38.482 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:50.709 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.531 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:50.758 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.571 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:08:50.798 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.590 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:50.817 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.594 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:50.810 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.602 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:50.830 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.604 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:50.820 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.635 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:08:50.862 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:38.637 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:08:50.852 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:44.038 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:44.150 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.149+0000 7fb1aa1ed640 10 monclient: get_monmap_and_config 2024-03-25T18:22:44.150 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.149+0000 7fb1aa1ed640 10 monclient: build_initial_monmap 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.149+0000 7fb1aa1ed640 10 monclient: monmap: 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:22:44.150469+0000 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:22:44.150469+0000 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:22:44.151 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.150+0000 7fb1aa1ed640 1 Processor -- start 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.150+0000 7fb1aa1ed640 1 -- start start 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.150+0000 7fb1aa1ed640 10 monclient: init 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a40a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a41430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a4142520 con 0x7fb1a413f710 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40b28a0 con 0x7fb1a41430c0 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40b2a30 con 0x7fb1a40a98f0 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:44.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.151+0000 7fb1aa1ed640 10 monclient(hunting): authenticate will time out at 8063.424867s 2024-03-25T18:22:44.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.154+0000 7fb1a2ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb1a413f710 0x7fb1a41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38428/0 (socket says 172.21.15.162:38428) 2024-03-25T18:22:44.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.154+0000 7fb1a2ffd640 1 -- 172.21.15.162:0/3243922767 learned_addr learned my addr 172.21.15.162:0/3243922767 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:44.172 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.171+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2191864864 0 0) 0x7fb1a4142520 con 0x7fb1a413f710 2024-03-25T18:22:44.172 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.171+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.172 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.171+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.172 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.171+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.172 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.171+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:22:44.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.171+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb18c003520 con 0x7fb1a413f710 2024-03-25T18:22:44.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.180+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1473709079 0 0) 0x7fb1a40b28a0 con 0x7fb1a41430c0 2024-03-25T18:22:44.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.180+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.180+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.180+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.180+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:22:44.182 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.180+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a4142520 con 0x7fb1a41430c0 2024-03-25T18:22:44.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.186+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698373748 0 0) 0x7fb1a40b2a30 con 0x7fb1a40a98f0 2024-03-25T18:22:44.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.186+0000 7fb1a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.186+0000 7fb1a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.186+0000 7fb1a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.186+0000 7fb1a27fc640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:22:44.187 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.186+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a40b28a0 con 0x7fb1a40a98f0 2024-03-25T18:22:44.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3436508373 0 0) 0x7fb1a4142520 con 0x7fb1a41430c0 2024-03-25T18:22:44.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 >> v1:172.21.15.162:6790/0 conn(0x7fb1a40a98f0 legacy=0x7fb1a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 >> v1:172.21.15.162:6789/0 conn(0x7fb1a413f710 legacy=0x7fb1a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 1 monclient: found mon.noname-c 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a40b44c0 con 0x7fb1a41430c0 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb19c0019f0 con 0x7fb1a41430c0 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1a27fc640 10 monclient: dump: 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:44.193 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1aa1ed640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:22:44.194 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.191+0000 7fb1aa1ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:44.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.206+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb19c001b50 con 0x7fb1a41430c0 2024-03-25T18:22:44.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.206+0000 7fb1a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:44.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.206+0000 7fb1a27fc640 1 -- 172.21.15.162:0/3243922767 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb19c001f60 con 0x7fb1a41430c0 2024-03-25T18:22:44.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.206+0000 7fb1aa1ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:44.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.206+0000 7fb1aa1ed640 4 set_mon_vals no callback set 2024-03-25T18:22:44.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.206+0000 7fb1a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.206+0000 7fb1a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.206+0000 7fb1a27fc640 10 monclient: dump: 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:44.208 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.207+0000 7fb1aa1ed640 10 monclient: shutdown 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.207+0000 7fb1aa1ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.207+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/3243922767 >> v1:172.21.15.191:6789/0 conn(0x7fb1a41430c0 legacy=0x7fb1a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.207+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/3243922767 shutdown_connections 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.207+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/3243922767 >> 172.21.15.162:0/3243922767 conn(0x7fb1a40b7ea0 msgr2=0x7fb1a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.207+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/3243922767 shutdown_connections 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.207+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/3243922767 wait complete. 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 10 monclient: build_initial_monmap 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 10 monclient: monmap: 2024-03-25T18:22:44.210 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 1 librados: starting msgr at 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 1 librados: starting objecter 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 1 Processor -- start 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 1 -- start start 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 1 librados: setting wanted keys 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 1 librados: calling monclient init 2024-03-25T18:22:44.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.208+0000 7fb1aa1ed640 10 monclient: init 2024-03-25T18:22:44.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:44.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:44.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a40a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:44.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a413f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:44.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a41430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:44.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a413b030 con 0x7fb1a40a98f0 2024-03-25T18:22:44.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a413b310 con 0x7fb1a413f710 2024-03-25T18:22:44.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a413b5f0 con 0x7fb1a41430c0 2024-03-25T18:22:44.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:44.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.209+0000 7fb1aa1ed640 10 monclient(hunting): authenticate will time out at 8063.483339s 2024-03-25T18:22:44.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.219+0000 7fb1a37fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fb1a40a98f0 0x7fb1a4141270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:38466/0 (socket says 172.21.15.162:38466) 2024-03-25T18:22:44.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.219+0000 7fb1a37fe640 1 -- 172.21.15.162:0/2723058091 learned_addr learned my addr 172.21.15.162:0/2723058091 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:44.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.221+0000 7fb18bfff640 10 client.?.objecter ms_handle_connect 0x7fb1a40a98f0 2024-03-25T18:22:44.222 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.221+0000 7fb18bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:44.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.223+0000 7fb18bfff640 10 client.?.objecter ms_handle_connect 0x7fb1a41430c0 2024-03-25T18:22:44.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.223+0000 7fb18bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:44.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.226+0000 7fb18bfff640 10 client.?.objecter ms_handle_connect 0x7fb1a413f710 2024-03-25T18:22:44.227 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.226+0000 7fb18bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:44.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.233+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578499075 0 0) 0x7fb1a413b030 con 0x7fb1a40a98f0 2024-03-25T18:22:44.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.233+0000 7fb18bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.233+0000 7fb18bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.233+0000 7fb18bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.233+0000 7fb18bfff640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:22:44.234 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.233+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190004eb0 con 0x7fb1a40a98f0 2024-03-25T18:22:44.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.240+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280879027 0 0) 0x7fb1a413b310 con 0x7fb1a413f710 2024-03-25T18:22:44.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.240+0000 7fb18bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.240+0000 7fb18bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.240+0000 7fb18bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.240+0000 7fb18bfff640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:22:44.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.240+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a413b030 con 0x7fb1a413f710 2024-03-25T18:22:44.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.246+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780735751 0 0) 0x7fb1a413b5f0 con 0x7fb1a41430c0 2024-03-25T18:22:44.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.246+0000 7fb18bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:44.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.246+0000 7fb18bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:44.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.246+0000 7fb18bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:44.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.246+0000 7fb18bfff640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:22:44.247 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.246+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1a413b310 con 0x7fb1a41430c0 2024-03-25T18:22:44.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.255+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2871434965 0 0) 0x7fb1a413b310 con 0x7fb1a41430c0 2024-03-25T18:22:44.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.255+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1a413b5f0 con 0x7fb1a41430c0 2024-03-25T18:22:44.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.255+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb19c001e60 con 0x7fb1a41430c0 2024-03-25T18:22:44.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.255+0000 7fb18bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.255+0000 7fb18bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:44.256 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.255+0000 7fb18bfff640 10 monclient(hunting): dump: 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:44.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.255+0000 7fb18bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:44.264 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3678285722 0 0) 0x7fb1a413b5f0 con 0x7fb1a41430c0 2024-03-25T18:22:44.264 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 >> v1:172.21.15.191:6789/0 conn(0x7fb1a413f710 legacy=0x7fb1a4139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.264 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 >> v1:172.21.15.162:6789/0 conn(0x7fb1a40a98f0 legacy=0x7fb1a4141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:44.264 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 1 monclient: found mon.c 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a40b7ad0 con 0x7fb1a41430c0 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 10 monclient: _finish_auth 0 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb18bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 5 monclient: authenticate success, global_id 4610 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 10 monclient: _renew_subs 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a413a430 con 0x7fb1a41430c0 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 10 monclient: _renew_subs 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a413bbf0 con 0x7fb1a41430c0 2024-03-25T18:22:44.265 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.263+0000 7fb1aa1ed640 1 librados: init done 2024-03-25T18:22:44.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.264+0000 7fb1aa1ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:44.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.264+0000 7fb1aa1ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:44.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.264+0000 7fb1aa1ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:44.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.264+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a40627b0 con 0x7fb1a41430c0 2024-03-25T18:22:44.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.567+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb19c002000 con 0x7fb1a41430c0 2024-03-25T18:22:44.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.567+0000 7fb18bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:44.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.567+0000 7fb1a1ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:44.577 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.576+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb19c003dc0 con 0x7fb1a41430c0 2024-03-25T18:22:44.577 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.576+0000 7fb18bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:44.577 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.576+0000 7fb18bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.576+0000 7fb18bfff640 10 monclient: dump: 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:44.578 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:44.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.577+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fb19c019d90 con 0x7fb1a41430c0 2024-03-25T18:22:44.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.577+0000 7fb1a37fe640 10 monclient: get_auth_request con 0x7fb19006de20 auth_method 0 2024-03-25T18:22:44.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.578+0000 7fb18bfff640 10 client.4610.objecter ms_handle_connect 0x7fb19006de20 2024-03-25T18:22:44.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.585+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fb19c053940 con 0x7fb1a41430c0 2024-03-25T18:22:44.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.585+0000 7fb18bfff640 10 client.4610.objecter ms_dispatch 0x7fb1a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:44.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.585+0000 7fb18bfff640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:44.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.585+0000 7fb18bfff640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:44.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.585+0000 7fb18bfff640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-25T18:22:44.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.585+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb19c04f280 con 0x7fb1a41430c0 2024-03-25T18:22:44.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.585+0000 7fb18bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:44.586 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.585+0000 7fb18bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:44.811 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.810+0000 7fb1aa1ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:44.811 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.810+0000 7fb1aa1ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:44.811 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.810+0000 7fb1aa1ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:22:44.812 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:44.810+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1a413ce30 con 0x7fb1a41430c0 2024-03-25T18:22:45.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.021+0000 7fb18bfff640 1 -- 172.21.15.162:0/2723058091 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fb19c051380 con 0x7fb1a41430c0 2024-03-25T18:22:45.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.021+0000 7fb18bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:45.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.021+0000 7fb18bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:45.023 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:22:45.023 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 10 librados: watch_flush enter 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 10 librados: watch_flush exit 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 10 client.4610.objecter successfully canceled tick 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 >> v1:172.21.15.191:6800/1552703795 conn(0x7fb19006de20 legacy=0x7fb1900702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 10 monclient: shutdown 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 >> v1:172.21.15.162:6790/0 conn(0x7fb1a41430c0 legacy=0x7fb1a41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:45.026 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 shutdown_connections 2024-03-25T18:22:45.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.024+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 >> 172.21.15.162:0/2723058091 conn(0x7fb1a40b7ea0 msgr2=0x7fb1a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:45.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.025+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 shutdown_connections 2024-03-25T18:22:45.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.025+0000 7fb1aa1ed640 1 -- 172.21.15.162:0/2723058091 wait complete. 2024-03-25T18:22:45.027 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:45.025+0000 7fb1aa1ed640 1 librados: shutdown 2024-03-25T18:22:45.037 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:51.844 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:51.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.960+0000 7fc36a2e4640 10 monclient: get_monmap_and_config 2024-03-25T18:22:51.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.961+0000 7fc36a2e4640 10 monclient: build_initial_monmap 2024-03-25T18:22:51.962 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.961+0000 7fc36a2e4640 10 monclient: monmap: 2024-03-25T18:22:51.962 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:51.962 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:51.962 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:22:51.962055+0000 2024-03-25T18:22:51.962 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:22:51.962055+0000 2024-03-25T18:22:51.962 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:51.963 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:51.963 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:51.963 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:22:51.963 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:22:51.963 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:51.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.961+0000 7fc36a2e4640 1 Processor -- start 2024-03-25T18:22:51.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.961+0000 7fc36a2e4640 1 -- start start 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.961+0000 7fc36a2e4640 10 monclient: init 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.962+0000 7fc36a2e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.962+0000 7fc36a2e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.962+0000 7fc36a2e4640 10 monclient(hunting): picked mon.noname-a con 0x7fc3640a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.962+0000 7fc36a2e4640 10 monclient(hunting): picked mon.noname-b con 0x7fc36413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.962+0000 7fc36a2e4640 10 monclient(hunting): picked mon.noname-c con 0x7fc3641430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.963+0000 7fc36a2e4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc364142520 con 0x7fc3640a98f0 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.963+0000 7fc36a2e4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3640b28a0 con 0x7fc3641430c0 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.963+0000 7fc36a2e4640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3640b2a30 con 0x7fc36413f710 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.963+0000 7fc36a2e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:51.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.963+0000 7fc36a2e4640 10 monclient(hunting): authenticate will time out at 8071.236745s 2024-03-25T18:22:51.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.977+0000 7fc368881640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc3641430c0 0x7fc36414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:59696/0 (socket says 172.21.15.162:59696) 2024-03-25T18:22:51.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.977+0000 7fc368881640 1 -- 172.21.15.162:0/3277585676 learned_addr learned my addr 172.21.15.162:0/3277585676 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:51.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.987+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098316262 0 0) 0x7fc3640b28a0 con 0x7fc3641430c0 2024-03-25T18:22:51.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.987+0000 7fc362ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:51.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.987+0000 7fc362ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:51.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.987+0000 7fc362ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:51.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.987+0000 7fc362ffd640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:22:51.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:51.987+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34c003520 con 0x7fc3641430c0 2024-03-25T18:22:52.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.001+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920588635 0 0) 0x7fc3640b2a30 con 0x7fc36413f710 2024-03-25T18:22:52.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.001+0000 7fc362ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:52.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.001+0000 7fc362ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:52.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.001+0000 7fc362ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:52.002 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.001+0000 7fc362ffd640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:22:52.003 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.001+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3640b28a0 con 0x7fc36413f710 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1638165775 0 0) 0x7fc34c003520 con 0x7fc3641430c0 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 >> v1:172.21.15.162:6790/0 conn(0x7fc36413f710 legacy=0x7fc3641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 >> v1:172.21.15.162:6789/0 conn(0x7fc3640a98f0 legacy=0x7fc3640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 1 monclient: found mon.noname-c 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3640b44c0 con 0x7fc3641430c0 2024-03-25T18:22:52.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc35c001a10 con 0x7fc3641430c0 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc36a2e4640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc362ffd640 10 monclient: dump: 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:52.007 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:52.008 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:52.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.004+0000 7fc36a2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:52.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc35c001b70 con 0x7fc3641430c0 2024-03-25T18:22:52.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc362ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:52.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc362ffd640 1 -- 172.21.15.162:0/3277585676 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc35c001f80 con 0x7fc3641430c0 2024-03-25T18:22:52.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc362ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:52.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc362ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:52.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc362ffd640 10 monclient: dump: 2024-03-25T18:22:52.019 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:52.019 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc36a2e4640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:52.020 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc36a2e4640 4 set_mon_vals no callback set 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc36a2e4640 10 monclient: shutdown 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc36a2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc36a2e4640 1 -- 172.21.15.162:0/3277585676 >> v1:172.21.15.191:6789/0 conn(0x7fc3641430c0 legacy=0x7fc36414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc36a2e4640 1 -- 172.21.15.162:0/3277585676 shutdown_connections 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.018+0000 7fc36a2e4640 1 -- 172.21.15.162:0/3277585676 >> 172.21.15.162:0/3277585676 conn(0x7fc3640b7ea0 msgr2=0x7fc3640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.019+0000 7fc36a2e4640 1 -- 172.21.15.162:0/3277585676 shutdown_connections 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.019+0000 7fc36a2e4640 1 -- 172.21.15.162:0/3277585676 wait complete. 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.019+0000 7fc36a2e4640 10 monclient: build_initial_monmap 2024-03-25T18:22:52.023 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.020+0000 7fc36a2e4640 10 monclient: monmap: 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.020+0000 7fc36a2e4640 1 librados: starting msgr at 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.020+0000 7fc36a2e4640 1 librados: starting objecter 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.020+0000 7fc36a2e4640 1 Processor -- start 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.020+0000 7fc36a2e4640 1 -- start start 2024-03-25T18:22:52.024 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.020+0000 7fc36a2e4640 1 librados: setting wanted keys 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.020+0000 7fc36a2e4640 1 librados: calling monclient init 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.020+0000 7fc36a2e4640 10 monclient: init 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 10 monclient(hunting): picked mon.noname-b con 0x7fc3640a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 10 monclient(hunting): picked mon.noname-a con 0x7fc36413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 10 monclient(hunting): picked mon.noname-c con 0x7fc3641430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3641354c0 con 0x7fc36413f710 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3641319b0 con 0x7fc3640a98f0 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc364135070 con 0x7fc3641430c0 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:52.025 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.021+0000 7fc36a2e4640 10 monclient(hunting): authenticate will time out at 8071.295410s 2024-03-25T18:22:52.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.032+0000 7fc3637fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc36413f710 0x7fc364130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:43728/0 (socket says 172.21.15.162:43728) 2024-03-25T18:22:52.033 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.032+0000 7fc3637fe640 1 -- 172.21.15.162:0/206981405 learned_addr learned my addr 172.21.15.162:0/206981405 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:52.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.035+0000 7fc360ff9640 10 client.?.objecter ms_handle_connect 0x7fc36413f710 2024-03-25T18:22:52.036 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.035+0000 7fc360ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:52.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.039+0000 7fc360ff9640 10 client.?.objecter ms_handle_connect 0x7fc3640a98f0 2024-03-25T18:22:52.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.039+0000 7fc360ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:52.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.046+0000 7fc360ff9640 10 client.?.objecter ms_handle_connect 0x7fc3641430c0 2024-03-25T18:22:52.047 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.046+0000 7fc360ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:52.049 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.047+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739589846 0 0) 0x7fc3641354c0 con 0x7fc36413f710 2024-03-25T18:22:52.049 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.048+0000 7fc360ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:52.049 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.048+0000 7fc360ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:52.049 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.048+0000 7fc360ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:52.049 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.048+0000 7fc360ff9640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:22:52.049 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.048+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc350004eb0 con 0x7fc36413f710 2024-03-25T18:22:52.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.057+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468236543 0 0) 0x7fc3641319b0 con 0x7fc3640a98f0 2024-03-25T18:22:52.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.057+0000 7fc360ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:52.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.057+0000 7fc360ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:52.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.057+0000 7fc360ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:52.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.057+0000 7fc360ff9640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:22:52.059 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.057+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3641354c0 con 0x7fc3640a98f0 2024-03-25T18:22:52.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.066+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2613500886 0 0) 0x7fc364135070 con 0x7fc3641430c0 2024-03-25T18:22:52.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.066+0000 7fc360ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:52.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.066+0000 7fc360ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:52.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.066+0000 7fc360ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:52.068 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.066+0000 7fc360ff9640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:22:52.068 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.066+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3641319b0 con 0x7fc3641430c0 2024-03-25T18:22:52.079 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.078+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 457121851 0 0) 0x7fc350004eb0 con 0x7fc36413f710 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.078+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc364135070 con 0x7fc36413f710 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.078+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc358001a10 con 0x7fc36413f710 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.078+0000 7fc360ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.078+0000 7fc360ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.078+0000 7fc360ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:52.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.078+0000 7fc360ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.086+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2360177368 0 0) 0x7fc3641354c0 con 0x7fc3640a98f0 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.087+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc350004eb0 con 0x7fc3640a98f0 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.087+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc3540018a0 con 0x7fc3640a98f0 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.087+0000 7fc360ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.087+0000 7fc360ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.087+0000 7fc360ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:52.088 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:52.089 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:52.089 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:52.089 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:52.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.094+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1052086032 0 0) 0x7fc3641319b0 con 0x7fc3641430c0 2024-03-25T18:22:52.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.094+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3641354c0 con 0x7fc3641430c0 2024-03-25T18:22:52.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.094+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc35c001e80 con 0x7fc3641430c0 2024-03-25T18:22:52.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.094+0000 7fc360ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:52.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.094+0000 7fc360ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:52.095 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.094+0000 7fc360ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:52.095 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:52.095 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:52.096 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.096 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.096 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:52.096 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:52.096 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:52.096 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:52.096 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:52.096 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1689646927 0 0) 0x7fc364135070 con 0x7fc36413f710 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 >> v1:172.21.15.162:6790/0 conn(0x7fc3641430c0 legacy=0x7fc3641312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 >> v1:172.21.15.191:6789/0 conn(0x7fc3640a98f0 legacy=0x7fc3641d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 1 monclient: found mon.a 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc364135200 con 0x7fc36413f710 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc360ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 5 monclient: authenticate success, global_id 4635 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 10 monclient: _renew_subs 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:22:52.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3640b7010 con 0x7fc36413f710 2024-03-25T18:22:52.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:52.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 10 monclient: _renew_subs 2024-03-25T18:22:52.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:22:52.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36413b890 con 0x7fc36413f710 2024-03-25T18:22:52.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.101+0000 7fc36a2e4640 1 librados: init done 2024-03-25T18:22:52.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.102+0000 7fc36a2e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:52.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.102+0000 7fc36a2e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:52.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.102+0000 7fc36a2e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:22:52.105 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.102+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc364137160 con 0x7fc36413f710 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.415+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc358001c50 con 0x7fc36413f710 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.415+0000 7fc360ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.415+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc358003a60 con 0x7fc36413f710 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.415+0000 7fc360ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.415+0000 7fc360ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.415+0000 7fc360ff9640 10 monclient: dump: 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.417 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:52.418 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:52.418 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:52.418 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:52.418 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:52.418 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:52.418 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:52.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.415+0000 7fc3627fc640 4 set_mon_vals no callback set 2024-03-25T18:22:52.418 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.416+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fc358003c80 con 0x7fc36413f710 2024-03-25T18:22:52.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.416+0000 7fc363fff640 10 monclient: get_auth_request con 0x7fc35006e0f0 auth_method 0 2024-03-25T18:22:52.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.417+0000 7fc360ff9640 10 client.4635.objecter ms_handle_connect 0x7fc35006e0f0 2024-03-25T18:22:52.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.424+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fc3580548b0 con 0x7fc36413f710 2024-03-25T18:22:52.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.424+0000 7fc360ff9640 10 client.4635.objecter ms_dispatch 0x7fc364000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:52.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.424+0000 7fc360ff9640 3 client.4635.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:52.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.424+0000 7fc360ff9640 3 client.4635.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:52.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.424+0000 7fc360ff9640 20 client.4635.objecter dump_active .. 0 homeless 2024-03-25T18:22:52.425 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.424+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3580503c0 con 0x7fc36413f710 2024-03-25T18:22:52.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.424+0000 7fc360ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:52.426 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.424+0000 7fc360ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:52.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.689+0000 7fc36a2e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:52.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.689+0000 7fc36a2e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:52.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.689+0000 7fc36a2e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:22:52.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.689+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc364135670 con 0x7fc36413f710 2024-03-25T18:22:52.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.714+0000 7fc360ff9640 1 -- 172.21.15.162:0/206981405 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fc364135670 con 0x7fc36413f710 2024-03-25T18:22:52.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.714+0000 7fc360ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:52.715 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.714+0000 7fc360ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:52.716 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:22:52.716 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 10 librados: watch_flush enter 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 10 librados: watch_flush exit 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 20 client.4635.objecter shutdown clearing up homeless session... 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 10 client.4635.objecter successfully canceled tick 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 >> v1:172.21.15.191:6800/1552703795 conn(0x7fc35006e0f0 legacy=0x7fc3500705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 10 monclient: shutdown 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 >> v1:172.21.15.162:6789/0 conn(0x7fc36413f710 legacy=0x7fc364130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 shutdown_connections 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 >> 172.21.15.162:0/206981405 conn(0x7fc3640b7ea0 msgr2=0x7fc36413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:52.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 shutdown_connections 2024-03-25T18:22:52.719 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 1 -- 172.21.15.162:0/206981405 wait complete. 2024-03-25T18:22:52.719 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:52.716+0000 7fc36a2e4640 1 librados: shutdown 2024-03-25T18:22:52.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:58.558 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:10.785 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:58.605 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:10.820 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:58.607 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:10.835 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:58.615 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:10.831 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:58.654 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:10.881 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:58.654 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:10.870 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:58.681 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:10.909 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:58.688 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:10.916 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:58.716 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:10.944 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:59.535 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:59.652 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.651+0000 7fcdfe397640 10 monclient: get_monmap_and_config 2024-03-25T18:22:59.652 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.651+0000 7fcdfe397640 10 monclient: build_initial_monmap 2024-03-25T18:22:59.652 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.651+0000 7fcdfe397640 10 monclient: monmap: 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:22:59.652426+0000 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:22:59.652426+0000 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:22:59.653 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:59.654 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.652+0000 7fcdfe397640 1 Processor -- start 2024-03-25T18:22:59.654 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.652+0000 7fcdfe397640 1 -- start start 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.652+0000 7fcdfe397640 10 monclient: init 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf80a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf813f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf81430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf8142520 con 0x7fcdf81430c0 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf80b28a0 con 0x7fcdf813f710 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf80b2a30 con 0x7fcdf80a98f0 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:59.655 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.653+0000 7fcdfe397640 10 monclient(hunting): authenticate will time out at 8078.926913s 2024-03-25T18:22:59.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.660+0000 7fcdf7fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fcdf80a98f0 0x7fcdf80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:41144/0 (socket says 172.21.15.162:41144) 2024-03-25T18:22:59.661 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.660+0000 7fcdf7fff640 1 -- 172.21.15.162:0/3323075634 learned_addr learned my addr 172.21.15.162:0/3323075634 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:59.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.670+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666554486 0 0) 0x7fcdf80b2a30 con 0x7fcdf80a98f0 2024-03-25T18:22:59.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.670+0000 7fcdf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.670+0000 7fcdf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.670+0000 7fcdf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.670+0000 7fcdf6ffd640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:22:59.672 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.670+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcde4003520 con 0x7fcdf80a98f0 2024-03-25T18:22:59.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.684+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841961023 0 0) 0x7fcdf8142520 con 0x7fcdf81430c0 2024-03-25T18:22:59.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.684+0000 7fcdf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.685 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.684+0000 7fcdf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.684+0000 7fcdf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.684+0000 7fcdf6ffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:22:59.686 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.684+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf80b2a30 con 0x7fcdf81430c0 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3434707925 0 0) 0x7fcde4003520 con 0x7fcdf80a98f0 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 >> v1:172.21.15.191:6789/0 conn(0x7fcdf813f710 legacy=0x7fcdf81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 >> v1:172.21.15.162:6789/0 conn(0x7fcdf81430c0 legacy=0x7fcdf814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 1 monclient: found mon.noname-b 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf80b44c0 con 0x7fcdf80a98f0 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcde8001a10 con 0x7fcdf80a98f0 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.693 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdf6ffd640 10 monclient: dump: 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdfe397640 5 monclient: authenticate success, global_id 4619 2024-03-25T18:22:59.694 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.691+0000 7fcdfe397640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.701+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde8001b70 con 0x7fcdf80a98f0 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.701+0000 7fcdf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.701+0000 7fcdf6ffd640 1 -- 172.21.15.162:0/3323075634 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcde8001f80 con 0x7fcdf80a98f0 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.701+0000 7fcdf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.701+0000 7fcdf6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.701+0000 7fcdf6ffd640 10 monclient: dump: 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:59.708 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.701+0000 7fcdfe397640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.701+0000 7fcdfe397640 4 set_mon_vals no callback set 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.702+0000 7fcdfe397640 10 monclient: shutdown 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.702+0000 7fcdfe397640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.702+0000 7fcdfe397640 1 -- 172.21.15.162:0/3323075634 >> v1:172.21.15.162:6790/0 conn(0x7fcdf80a98f0 legacy=0x7fcdf80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.702+0000 7fcdfe397640 1 -- 172.21.15.162:0/3323075634 shutdown_connections 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.702+0000 7fcdfe397640 1 -- 172.21.15.162:0/3323075634 >> 172.21.15.162:0/3323075634 conn(0x7fcdf80b7ea0 msgr2=0x7fcdf80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:59.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.702+0000 7fcdfe397640 1 -- 172.21.15.162:0/3323075634 shutdown_connections 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.702+0000 7fcdfe397640 1 -- 172.21.15.162:0/3323075634 wait complete. 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.703+0000 7fcdfe397640 10 monclient: build_initial_monmap 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.703+0000 7fcdfe397640 10 monclient: monmap: 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.704+0000 7fcdfe397640 1 librados: starting msgr at 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.704+0000 7fcdfe397640 1 librados: starting objecter 2024-03-25T18:22:59.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.704+0000 7fcdfe397640 1 Processor -- start 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.704+0000 7fcdfe397640 1 -- start start 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.704+0000 7fcdfe397640 1 librados: setting wanted keys 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.704+0000 7fcdfe397640 1 librados: calling monclient init 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.704+0000 7fcdfe397640 10 monclient: init 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf80a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf813f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf81430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf81370a0 con 0x7fcdf81430c0 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf8135090 con 0x7fcdf80a98f0 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf8135220 con 0x7fcdf813f710 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:59.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.705+0000 7fcdfe397640 10 monclient(hunting): authenticate will time out at 8078.979244s 2024-03-25T18:22:59.717 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.716+0000 7fcdf77fe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fcdf813f710 0x7fcdf81d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:41176/0 (socket says 172.21.15.162:41176) 2024-03-25T18:22:59.717 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.716+0000 7fcdf77fe640 1 -- 172.21.15.162:0/1659907109 learned_addr learned my addr 172.21.15.162:0/1659907109 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:22:59.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.717+0000 7fcdf4ff9640 10 client.?.objecter ms_handle_connect 0x7fcdf813f710 2024-03-25T18:22:59.718 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.717+0000 7fcdf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:59.725 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.724+0000 7fcdf4ff9640 10 client.?.objecter ms_handle_connect 0x7fcdf80a98f0 2024-03-25T18:22:59.725 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.724+0000 7fcdf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:59.725 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.724+0000 7fcdf4ff9640 10 client.?.objecter ms_handle_connect 0x7fcdf81430c0 2024-03-25T18:22:59.726 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.724+0000 7fcdf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:59.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.736+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660402544 0 0) 0x7fcdf8135220 con 0x7fcdf813f710 2024-03-25T18:22:59.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.736+0000 7fcdf4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.736+0000 7fcdf4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.736+0000 7fcdf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.736+0000 7fcdf4ff9640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:22:59.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.736+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd8004eb0 con 0x7fcdf813f710 2024-03-25T18:22:59.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.739+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381374111 0 0) 0x7fcdf8135090 con 0x7fcdf80a98f0 2024-03-25T18:22:59.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.739+0000 7fcdf4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.739+0000 7fcdf4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.739+0000 7fcdf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.739+0000 7fcdf4ff9640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:22:59.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.739+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf8135220 con 0x7fcdf80a98f0 2024-03-25T18:22:59.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.742+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267799644 0 0) 0x7fcdf81370a0 con 0x7fcdf81430c0 2024-03-25T18:22:59.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.742+0000 7fcdf4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:59.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.742+0000 7fcdf4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:59.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.742+0000 7fcdf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:59.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.742+0000 7fcdf4ff9640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:22:59.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.742+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdf8135090 con 0x7fcdf81430c0 2024-03-25T18:22:59.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.756+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 151095146 0 0) 0x7fcdf8135220 con 0x7fcdf80a98f0 2024-03-25T18:22:59.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.756+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdf81370a0 con 0x7fcdf80a98f0 2024-03-25T18:22:59.758 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.757+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcde8001e80 con 0x7fcdf80a98f0 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.757+0000 7fcdf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.757+0000 7fcdf4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.757+0000 7fcdf4ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:59.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.757+0000 7fcdf4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:59.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.767+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2855616902 0 0) 0x7fcdf8135090 con 0x7fcdf81430c0 2024-03-25T18:22:59.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.767+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdf8135220 con 0x7fcdf81430c0 2024-03-25T18:22:59.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.767+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcdf00017d0 con 0x7fcdf81430c0 2024-03-25T18:22:59.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.767+0000 7fcdf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.767+0000 7fcdf4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.767+0000 7fcdf4ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:59.769 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.773+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2169501874 0 0) 0x7fcdd8004eb0 con 0x7fcdf813f710 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.773+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdf8135090 con 0x7fcdf813f710 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.773+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcde0001a30 con 0x7fcdf813f710 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.773+0000 7fcdf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.773+0000 7fcdf4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.773+0000 7fcdf4ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:22:59.775 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:22:59.776 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:22:59.776 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:22:59.776 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:22:59.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.783+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2581346161 0 0) 0x7fcdf81370a0 con 0x7fcdf80a98f0 2024-03-25T18:22:59.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.783+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 >> v1:172.21.15.162:6790/0 conn(0x7fcdf813f710 legacy=0x7fcdf81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.783+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 >> v1:172.21.15.162:6789/0 conn(0x7fcdf81430c0 legacy=0x7fcdf81d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.783+0000 7fcdf4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:59.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.783+0000 7fcdf4ff9640 1 monclient: found mon.b 2024-03-25T18:22:59.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.783+0000 7fcdf4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.783+0000 7fcdf4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.783+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf81d5000 con 0x7fcdf80a98f0 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdf4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdf4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdf4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 10 monclient: _renew_subs 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf81d52b0 con 0x7fcdf80a98f0 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 10 monclient: _renew_subs 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf8135910 con 0x7fcdf80a98f0 2024-03-25T18:22:59.786 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.784+0000 7fcdfe397640 1 librados: init done 2024-03-25T18:22:59.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.785+0000 7fcdfe397640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:59.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.785+0000 7fcdfe397640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:59.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.785+0000 7fcdfe397640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:22:59.787 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:22:59.785+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf80627b0 con 0x7fcdf80a98f0 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.058+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde8002020 con 0x7fcdf80a98f0 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.058+0000 7fcdf4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.058+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcde8004fb0 con 0x7fcdf80a98f0 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.058+0000 7fcdf4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.058+0000 7fcdf4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.058+0000 7fcdf4ff9640 10 monclient: dump: 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:00.060 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:00.061 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:00.061 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:00.061 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:00.061 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:00.061 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:00.061 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:00.061 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.058+0000 7fcdf5ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:00.061 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.059+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fcde801af80 con 0x7fcdf80a98f0 2024-03-25T18:23:00.062 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.060+0000 7fcdf77fe640 10 monclient: get_auth_request con 0x7fcdd806e110 auth_method 0 2024-03-25T18:23:00.062 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.060+0000 7fcdf4ff9640 10 client.4621.objecter ms_handle_connect 0x7fcdd806e110 2024-03-25T18:23:00.065 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.064+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fcde8054b80 con 0x7fcdf80a98f0 2024-03-25T18:23:00.065 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.064+0000 7fcdf4ff9640 10 client.4621.objecter ms_dispatch 0x7fcdf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:00.065 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.064+0000 7fcdf4ff9640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:00.065 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.064+0000 7fcdf4ff9640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:00.066 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.064+0000 7fcdf4ff9640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T18:23:00.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.066+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcde80504c0 con 0x7fcdf80a98f0 2024-03-25T18:23:00.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.066+0000 7fcdf4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:00.067 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.066+0000 7fcdf4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:00.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.327+0000 7fcdfe397640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:00.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.327+0000 7fcdfe397640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:00.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.327+0000 7fcdfe397640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:00.328 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.327+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdf813bfe0 con 0x7fcdf80a98f0 2024-03-25T18:23:00.369 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.368+0000 7fcdf4ff9640 1 -- 172.21.15.162:0/1659907109 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fcde80525c0 con 0x7fcdf80a98f0 2024-03-25T18:23:00.369 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.368+0000 7fcdf4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:00.369 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.368+0000 7fcdf4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:00.369 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:23:00.369 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:00.371 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 10 librados: watch_flush enter 2024-03-25T18:23:00.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 10 librados: watch_flush exit 2024-03-25T18:23:00.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T18:23:00.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 10 client.4621.objecter successfully canceled tick 2024-03-25T18:23:00.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 >> v1:172.21.15.191:6800/1552703795 conn(0x7fcdd806e110 legacy=0x7fcdd80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:00.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 10 monclient: shutdown 2024-03-25T18:23:00.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:00.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 >> v1:172.21.15.191:6789/0 conn(0x7fcdf80a98f0 legacy=0x7fcdf81dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:00.372 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.370+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 shutdown_connections 2024-03-25T18:23:00.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.371+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 >> 172.21.15.162:0/1659907109 conn(0x7fcdf80b7ea0 msgr2=0x7fcdf8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:00.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.371+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 shutdown_connections 2024-03-25T18:23:00.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.371+0000 7fcdfe397640 1 -- 172.21.15.162:0/1659907109 wait complete. 2024-03-25T18:23:00.373 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:00.371+0000 7fcdfe397640 1 librados: shutdown 2024-03-25T18:23:00.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:07.191 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:07.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.308+0000 7fa7840f1640 10 monclient: get_monmap_and_config 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.308+0000 7fa7840f1640 10 monclient: build_initial_monmap 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.308+0000 7fa7840f1640 10 monclient: monmap: 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:23:07.309968+0000 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:23:07.309968+0000 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:23:07.310 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:07.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.309+0000 7fa7840f1640 1 Processor -- start 2024-03-25T18:23:07.311 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.309+0000 7fa7840f1640 1 -- start start 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.309+0000 7fa7840f1640 10 monclient: init 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 10 monclient(hunting): picked mon.noname-c con 0x7fa77c0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 10 monclient(hunting): picked mon.noname-a con 0x7fa77c0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 10 monclient(hunting): picked mon.noname-b con 0x7fa77c14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0af940 con 0x7fa77c0b1730 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0b0a70 con 0x7fa77c0b4d60 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0b0d50 con 0x7fa77c14be90 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:07.312 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.310+0000 7fa7840f1640 10 monclient(hunting): authenticate will time out at 8086.584202s 2024-03-25T18:23:07.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.322+0000 7fa78268e640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fa77c14be90 0x7fa77c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:39954/0 (socket says 172.21.15.162:39954) 2024-03-25T18:23:07.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.322+0000 7fa78268e640 1 -- 172.21.15.162:0/109611626 learned_addr learned my addr 172.21.15.162:0/109611626 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:07.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.333+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220934630 0 0) 0x7fa77c0b0d50 con 0x7fa77c14be90 2024-03-25T18:23:07.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.333+0000 7fa780e8b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.333+0000 7fa780e8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.333+0000 7fa780e8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.333+0000 7fa780e8b640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:23:07.335 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.333+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa768003520 con 0x7fa77c14be90 2024-03-25T18:23:07.336 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.335+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977609050 0 0) 0x7fa77c0b0a70 con 0x7fa77c0b4d60 2024-03-25T18:23:07.336 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.335+0000 7fa780e8b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.336 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.335+0000 7fa780e8b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.336 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.335+0000 7fa780e8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.336 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.335+0000 7fa780e8b640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:23:07.337 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.335+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77c0b0d50 con 0x7fa77c0b4d60 2024-03-25T18:23:07.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1176615847 0 0) 0x7fa77c0b0d50 con 0x7fa77c0b4d60 2024-03-25T18:23:07.346 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 >> v1:172.21.15.162:6790/0 conn(0x7fa77c14be90 legacy=0x7fa77c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 >> v1:172.21.15.162:6789/0 conn(0x7fa77c0b1730 legacy=0x7fa77c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 10 monclient: _finish_hunting 0 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 1 monclient: found mon.noname-c 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c0b1030 con 0x7fa77c0b4d60 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 10 monclient: _finish_auth 0 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 10 monclient: _check_auth_tickets 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa778001670 con 0x7fa77c0b4d60 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa780e8b640 10 monclient: dump: 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:07.347 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa7840f1640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:23:07.348 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.345+0000 7fa7840f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:07.357 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.356+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7780017d0 con 0x7fa77c0b4d60 2024-03-25T18:23:07.357 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.356+0000 7fa780e8b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:07.357 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.356+0000 7fa780e8b640 1 -- 172.21.15.162:0/109611626 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa778001ba0 con 0x7fa77c0b4d60 2024-03-25T18:23:07.357 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.356+0000 7fa780e8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.357 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.356+0000 7fa780e8b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:07.357 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.356+0000 7fa780e8b640 10 monclient: dump: 2024-03-25T18:23:07.357 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.356+0000 7fa7840f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:07.358 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.356+0000 7fa7840f1640 4 set_mon_vals no callback set 2024-03-25T18:23:07.359 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.357+0000 7fa7840f1640 10 monclient: shutdown 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.357+0000 7fa7840f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.357+0000 7fa7840f1640 1 -- 172.21.15.162:0/109611626 >> v1:172.21.15.191:6789/0 conn(0x7fa77c0b4d60 legacy=0x7fa77c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.357+0000 7fa7840f1640 1 -- 172.21.15.162:0/109611626 shutdown_connections 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.357+0000 7fa7840f1640 1 -- 172.21.15.162:0/109611626 >> 172.21.15.162:0/109611626 conn(0x7fa77c0617a0 msgr2=0x7fa77c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.357+0000 7fa7840f1640 1 -- 172.21.15.162:0/109611626 shutdown_connections 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.357+0000 7fa7840f1640 1 -- 172.21.15.162:0/109611626 wait complete. 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 10 monclient: build_initial_monmap 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 10 monclient: monmap: 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:23:07.360 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 1 librados: starting msgr at 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 1 librados: starting objecter 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 1 Processor -- start 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 1 -- start start 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 1 librados: setting wanted keys 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 1 librados: calling monclient init 2024-03-25T18:23:07.361 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.358+0000 7fa7840f1640 10 monclient: init 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 10 monclient(hunting): picked mon.noname-c con 0x7fa77c0b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 10 monclient(hunting): picked mon.noname-a con 0x7fa77c0b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 10 monclient(hunting): picked mon.noname-b con 0x7fa77c14be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0a9d70 con 0x7fa77c0b4d60 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0aac20 con 0x7fa77c14be90 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c0aaf00 con 0x7fa77c0b1730 2024-03-25T18:23:07.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:07.363 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.359+0000 7fa7840f1640 10 monclient(hunting): authenticate will time out at 8086.633137s 2024-03-25T18:23:07.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.363+0000 7fa78168c640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fa77c0b4d60 0x7fa77c140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:49490/0 (socket says 172.21.15.162:49490) 2024-03-25T18:23:07.364 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.363+0000 7fa78168c640 1 -- 172.21.15.162:0/1006828045 learned_addr learned my addr 172.21.15.162:0/1006828045 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:07.369 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.368+0000 7fa7727fc640 10 client.?.objecter ms_handle_connect 0x7fa77c14be90 2024-03-25T18:23:07.369 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.368+0000 7fa7727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:07.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.369+0000 7fa7727fc640 10 client.?.objecter ms_handle_connect 0x7fa77c0b4d60 2024-03-25T18:23:07.370 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.369+0000 7fa7727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:07.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.380+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925493387 0 0) 0x7fa77c0aac20 con 0x7fa77c14be90 2024-03-25T18:23:07.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.380+0000 7fa7727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.380+0000 7fa7727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.380+0000 7fa7727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.380+0000 7fa7727fc640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:23:07.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.380+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa760004eb0 con 0x7fa77c14be90 2024-03-25T18:23:07.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.384+0000 7fa7727fc640 10 client.?.objecter ms_handle_connect 0x7fa77c0b1730 2024-03-25T18:23:07.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.384+0000 7fa7727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:07.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.385+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858709246 0 0) 0x7fa77c0a9d70 con 0x7fa77c0b4d60 2024-03-25T18:23:07.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.385+0000 7fa7727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.385+0000 7fa7727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.385+0000 7fa7727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.385+0000 7fa7727fc640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:23:07.386 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.385+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77c0aac20 con 0x7fa77c0b4d60 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.401+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3425567459 0 0) 0x7fa760004eb0 con 0x7fa77c14be90 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.402+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa77c0a9d70 con 0x7fa77c14be90 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.402+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa774001b90 con 0x7fa77c14be90 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.402+0000 7fa7727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.402+0000 7fa7727fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.402+0000 7fa7727fc640 10 monclient(hunting): dump: 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.403 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:07.404 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:07.404 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:07.404 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:07.404 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:07.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.402+0000 7fa7727fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:07.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.403+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323633660 0 0) 0x7fa77c0aaf00 con 0x7fa77c0b1730 2024-03-25T18:23:07.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.404+0000 7fa7727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:07.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.404+0000 7fa7727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:07.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.404+0000 7fa7727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:07.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.404+0000 7fa7727fc640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:23:07.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.404+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa760004eb0 con 0x7fa77c0b1730 2024-03-25T18:23:07.414 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.413+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2711290976 0 0) 0x7fa77c0aac20 con 0x7fa77c0b4d60 2024-03-25T18:23:07.414 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.413+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa77c0aaf00 con 0x7fa77c0b4d60 2024-03-25T18:23:07.414 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.413+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa76c001970 con 0x7fa77c0b4d60 2024-03-25T18:23:07.414 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.413+0000 7fa7727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.414 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.413+0000 7fa7727fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:07.414 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.413+0000 7fa7727fc640 10 monclient(hunting): dump: 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:07.415 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:07.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 625459367 0 0) 0x7fa77c0a9d70 con 0x7fa77c14be90 2024-03-25T18:23:07.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 >> v1:172.21.15.162:6790/0 conn(0x7fa77c0b1730 legacy=0x7fa77c142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 >> v1:172.21.15.162:6789/0 conn(0x7fa77c0b4d60 legacy=0x7fa77c140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 1 monclient: found mon.b 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c0a9850 con 0x7fa77c14be90 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 10 monclient: _renew_subs 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c0a9470 con 0x7fa77c14be90 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 10 monclient: _renew_subs 2024-03-25T18:23:07.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:07.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c140c70 con 0x7fa77c14be90 2024-03-25T18:23:07.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.418+0000 7fa7840f1640 1 librados: init done 2024-03-25T18:23:07.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.419+0000 7fa7840f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:07.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.419+0000 7fa7840f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:07.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.419+0000 7fa7840f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:07.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.419+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c0a96b0 con 0x7fa77c14be90 2024-03-25T18:23:07.433 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.432+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa774001fe0 con 0x7fa77c14be90 2024-03-25T18:23:07.433 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.432+0000 7fa7727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:07.433 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.432+0000 7fa773fff640 4 set_mon_vals no callback set 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.473+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fa774004b60 con 0x7fa77c14be90 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.473+0000 7fa7727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.473+0000 7fa7727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.473+0000 7fa7727fc640 10 monclient: dump: 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:07.474 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:07.475 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:07.475 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:07.475 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:07.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.703+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fa774004d40 con 0x7fa77c14be90 2024-03-25T18:23:07.705 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.704+0000 7fa781e8d640 10 monclient: get_auth_request con 0x7fa76006df70 auth_method 0 2024-03-25T18:23:07.705 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.704+0000 7fa7727fc640 10 client.4627.objecter ms_handle_connect 0x7fa76006df70 2024-03-25T18:23:07.707 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.706+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7fa774054840 con 0x7fa77c14be90 2024-03-25T18:23:07.707 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.706+0000 7fa7727fc640 10 client.4627.objecter ms_dispatch 0x7fa77c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:07.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.706+0000 7fa7727fc640 3 client.4627.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:07.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.706+0000 7fa7727fc640 3 client.4627.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:07.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.706+0000 7fa7727fc640 20 client.4627.objecter dump_active .. 0 homeless 2024-03-25T18:23:07.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.710+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa774050180 con 0x7fa77c14be90 2024-03-25T18:23:07.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.710+0000 7fa7727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:07.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.710+0000 7fa7727fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:07.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.947+0000 7fa7840f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:07.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.947+0000 7fa7840f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:07.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.947+0000 7fa7840f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:07.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.947+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa77c0627b0 con 0x7fa77c14be90 2024-03-25T18:23:07.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.969+0000 7fa7727fc640 1 -- 172.21.15.162:0/1006828045 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7fa774052280 con 0x7fa77c14be90 2024-03-25T18:23:07.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.969+0000 7fa7727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:07.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.969+0000 7fa7727fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:07.971 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:23:07.971 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:07.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.972+0000 7fa7840f1640 10 librados: watch_flush enter 2024-03-25T18:23:07.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.972+0000 7fa7840f1640 10 librados: watch_flush exit 2024-03-25T18:23:07.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.972+0000 7fa7840f1640 20 client.4627.objecter shutdown clearing up homeless session... 2024-03-25T18:23:07.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.972+0000 7fa7840f1640 10 client.4627.objecter successfully canceled tick 2024-03-25T18:23:07.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 >> v1:172.21.15.191:6800/1552703795 conn(0x7fa76006df70 legacy=0x7fa760070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.974 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 10 monclient: shutdown 2024-03-25T18:23:07.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:07.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 >> v1:172.21.15.191:6789/0 conn(0x7fa77c14be90 legacy=0x7fa77c1e27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:07.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 shutdown_connections 2024-03-25T18:23:07.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 >> 172.21.15.162:0/1006828045 conn(0x7fa77c0617a0 msgr2=0x7fa77c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:07.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 shutdown_connections 2024-03-25T18:23:07.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 1 -- 172.21.15.162:0/1006828045 wait complete. 2024-03-25T18:23:07.975 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:07.973+0000 7fa7840f1640 1 librados: shutdown 2024-03-25T18:23:07.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:14.792 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.904+0000 7f6a0f5e2640 10 monclient: get_monmap_and_config 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.904+0000 7f6a0f5e2640 10 monclient: build_initial_monmap 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.904+0000 7f6a0f5e2640 10 monclient: monmap: 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:23:14.905133+0000 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:23:14.905133+0000 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:14.905 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:23:14.906 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:23:14.906 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.904+0000 7f6a0f5e2640 1 Processor -- start 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 1 -- start start 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 10 monclient: init 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 10 monclient(hunting): picked mon.noname-b con 0x7f6a080a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0813f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 10 monclient(hunting): picked mon.noname-a con 0x7f6a081430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a08142520 con 0x7f6a081430c0 2024-03-25T18:23:14.907 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a080b28a0 con 0x7f6a0813f710 2024-03-25T18:23:14.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a080b2a30 con 0x7f6a080a98f0 2024-03-25T18:23:14.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:14.908 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.905+0000 7f6a0f5e2640 10 monclient(hunting): authenticate will time out at 8094.179466s 2024-03-25T18:23:14.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.923+0000 7f6a0db7f640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f6a081430c0 0x7f6a0814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:48176/0 (socket says 172.21.15.162:48176) 2024-03-25T18:23:14.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.923+0000 7f6a0db7f640 1 -- 172.21.15.162:0/3479490006 learned_addr learned my addr 172.21.15.162:0/3479490006 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:14.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.935+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986700005 0 0) 0x7f6a08142520 con 0x7f6a081430c0 2024-03-25T18:23:14.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.935+0000 7f69fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:14.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.935+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:14.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.935+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:14.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.935+0000 7f69fffff640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:23:14.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.935+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f4003520 con 0x7f6a081430c0 2024-03-25T18:23:14.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.943+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449837713 0 0) 0x7f6a080b28a0 con 0x7f6a0813f710 2024-03-25T18:23:14.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.943+0000 7f69fffff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:14.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.943+0000 7f69fffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:14.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.943+0000 7f69fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:14.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.943+0000 7f69fffff640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:23:14.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.943+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a08142520 con 0x7f6a0813f710 2024-03-25T18:23:14.954 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1200852590 0 0) 0x7f69f4003520 con 0x7f6a081430c0 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 >> v1:172.21.15.162:6790/0 conn(0x7f6a080a98f0 legacy=0x7f6a080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 >> v1:172.21.15.191:6789/0 conn(0x7f6a0813f710 legacy=0x7f6a081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 1 monclient: found mon.noname-a 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a080b44c0 con 0x7f6a081430c0 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 10 monclient: _finish_auth 0 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6a00001a10 con 0x7f6a081430c0 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f69fffff640 10 monclient: dump: 2024-03-25T18:23:14.955 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f6a0f5e2640 5 monclient: authenticate success, global_id 4650 2024-03-25T18:23:14.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.953+0000 7f6a0f5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:14.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.964+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a00001b70 con 0x7f6a081430c0 2024-03-25T18:23:14.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.964+0000 7f69fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:14.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.964+0000 7f69fffff640 1 -- 172.21.15.162:0/3479490006 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6a00001f80 con 0x7f6a081430c0 2024-03-25T18:23:14.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.964+0000 7f69fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.964+0000 7f69fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.964+0000 7f69fffff640 10 monclient: dump: 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.964+0000 7f6a0f5e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:14.966 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.964+0000 7f6a0f5e2640 4 set_mon_vals no callback set 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.965+0000 7f6a0f5e2640 10 monclient: shutdown 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.965+0000 7f6a0f5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.965+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3479490006 >> v1:172.21.15.162:6789/0 conn(0x7f6a081430c0 legacy=0x7f6a0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.965+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3479490006 shutdown_connections 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.965+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3479490006 >> 172.21.15.162:0/3479490006 conn(0x7f6a080b7ea0 msgr2=0x7f6a080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.965+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3479490006 shutdown_connections 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.965+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3479490006 wait complete. 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.966+0000 7f6a0f5e2640 10 monclient: build_initial_monmap 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.966+0000 7f6a0f5e2640 10 monclient: monmap: 2024-03-25T18:23:14.968 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 1 librados: starting msgr at 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 1 librados: starting objecter 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 1 Processor -- start 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 1 -- start start 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 1 librados: setting wanted keys 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 1 librados: calling monclient init 2024-03-25T18:23:14.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 10 monclient: init 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 10 monclient(hunting): picked mon.noname-c con 0x7f6a080a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0813f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 10 monclient(hunting): picked mon.noname-a con 0x7f6a081430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.967+0000 7f6a0f5e2640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a08136c50 con 0x7f6a081430c0 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.968+0000 7f6a0f5e2640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a08136ec0 con 0x7f6a0813f710 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.968+0000 7f6a0f5e2640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a081317f0 con 0x7f6a080a98f0 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.968+0000 7f6a0f5e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.968+0000 7f6a0f5e2640 10 monclient(hunting): authenticate will time out at 8094.241682s 2024-03-25T18:23:14.971 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.970+0000 7f6a0d37e640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f6a080a98f0 0x7f6a081310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:60658/0 (socket says 172.21.15.162:60658) 2024-03-25T18:23:14.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.970+0000 7f6a0d37e640 1 -- 172.21.15.162:0/3798449423 learned_addr learned my addr 172.21.15.162:0/3798449423 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:14.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.971+0000 7f69fdffb640 10 client.?.objecter ms_handle_connect 0x7f6a080a98f0 2024-03-25T18:23:14.972 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.971+0000 7f69fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:14.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.980+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262226773 0 0) 0x7f6a081317f0 con 0x7f6a080a98f0 2024-03-25T18:23:14.981 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.980+0000 7f69fdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:14.982 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.980+0000 7f69fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:14.982 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.980+0000 7f69fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:14.982 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.980+0000 7f69fdffb640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:23:14.982 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.980+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec004eb0 con 0x7f6a080a98f0 2024-03-25T18:23:14.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.989+0000 7f69fdffb640 10 client.?.objecter ms_handle_connect 0x7f6a0813f710 2024-03-25T18:23:14.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.990+0000 7f69fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:15.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.999+0000 7f69fdffb640 10 client.?.objecter ms_handle_connect 0x7f6a081430c0 2024-03-25T18:23:15.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:14.999+0000 7f69fdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:15.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.000+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044187451 0 0) 0x7f6a08136ec0 con 0x7f6a0813f710 2024-03-25T18:23:15.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.000+0000 7f69fdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:15.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.000+0000 7f69fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:15.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.000+0000 7f69fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:15.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.000+0000 7f69fdffb640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:23:15.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.000+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a081317f0 con 0x7f6a0813f710 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.004+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2610137642 0 0) 0x7f69ec004eb0 con 0x7f6a080a98f0 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.004+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a08136ec0 con 0x7f6a080a98f0 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.004+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6a040019d0 con 0x7f6a080a98f0 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.004+0000 7f69fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.004+0000 7f69fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.004+0000 7f69fdffb640 10 monclient(hunting): dump: 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:15.005 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:15.006 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:15.006 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:15.006 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:15.006 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:15.006 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:15.006 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.004+0000 7f69fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:15.020 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3140740024 0 0) 0x7f6a08136ec0 con 0x7f6a080a98f0 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 >> v1:172.21.15.191:6789/0 conn(0x7f6a0813f710 legacy=0x7f6a081dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 >> v1:172.21.15.162:6789/0 conn(0x7f6a081430c0 legacy=0x7f6a081db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 1 monclient: found mon.c 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a08131980 con 0x7f6a080a98f0 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f69fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 5 monclient: authenticate success, global_id 4634 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 10 monclient: _renew_subs 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a08135070 con 0x7f6a080a98f0 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:15.021 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 10 monclient: _renew_subs 2024-03-25T18:23:15.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:23:15.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a080b7870 con 0x7f6a080a98f0 2024-03-25T18:23:15.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.019+0000 7f6a0f5e2640 1 librados: init done 2024-03-25T18:23:15.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.020+0000 7f6a0f5e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:15.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.020+0000 7f6a0f5e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:15.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.020+0000 7f6a0f5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:23:15.022 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.020+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a080627b0 con 0x7f6a080a98f0 2024-03-25T18:23:15.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.321+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a040035f0 con 0x7f6a080a98f0 2024-03-25T18:23:15.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.321+0000 7f69fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:15.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.321+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6a04003980 con 0x7f6a080a98f0 2024-03-25T18:23:15.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.321+0000 7f69fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:15.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.321+0000 7f69fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:15.322 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.321+0000 7f69fdffb640 10 monclient: dump: 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:15.323 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.321+0000 7f69ff7fe640 4 set_mon_vals no callback set 2024-03-25T18:23:15.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.322+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f6a04003ba0 con 0x7f6a080a98f0 2024-03-25T18:23:15.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.322+0000 7f6a0cb7d640 10 monclient: get_auth_request con 0x7f69ec06dd10 auth_method 0 2024-03-25T18:23:15.324 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.323+0000 7f69fdffb640 10 client.4634.objecter ms_handle_connect 0x7f69ec06dd10 2024-03-25T18:23:15.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.327+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f6a04054760 con 0x7f6a080a98f0 2024-03-25T18:23:15.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.327+0000 7f69fdffb640 10 client.4634.objecter ms_dispatch 0x7f6a08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:15.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.327+0000 7f69fdffb640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:15.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.327+0000 7f69fdffb640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:15.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.327+0000 7f69fdffb640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-25T18:23:15.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.327+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a04050270 con 0x7f6a080a98f0 2024-03-25T18:23:15.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.327+0000 7f69fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:15.329 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.327+0000 7f69fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:15.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.577+0000 7f6a0f5e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:15.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.577+0000 7f6a0f5e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:15.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.577+0000 7f6a0f5e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:23:15.578 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.577+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a080b7ac0 con 0x7f6a080a98f0 2024-03-25T18:23:15.604 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.603+0000 7f69fdffb640 1 -- 172.21.15.162:0/3798449423 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f6a040521a0 con 0x7f6a080a98f0 2024-03-25T18:23:15.604 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.603+0000 7f69fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:15.604 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.603+0000 7f69fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:15.605 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:23:15.605 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 10 librados: watch_flush enter 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 10 librados: watch_flush exit 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 10 client.4634.objecter successfully canceled tick 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 >> v1:172.21.15.191:6800/1552703795 conn(0x7f69ec06dd10 legacy=0x7f69ec0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 10 monclient: shutdown 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 >> v1:172.21.15.162:6790/0 conn(0x7f6a080a98f0 legacy=0x7f6a081310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 shutdown_connections 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 >> 172.21.15.162:0/3798449423 conn(0x7f6a080b7ea0 msgr2=0x7f6a0813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:15.608 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 shutdown_connections 2024-03-25T18:23:15.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 1 -- 172.21.15.162:0/3798449423 wait complete. 2024-03-25T18:23:15.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:15.606+0000 7f6a0f5e2640 1 librados: shutdown 2024-03-25T18:23:15.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:18.632 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:30.859 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:18.684 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:30.911 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:18.711 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:30.939 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:18.737 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:30.953 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:18.738 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:30.966 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:18.742 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:30.958 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:18.752 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:30.980 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:18.783 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:30.999 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:18.790 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:31.017 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:22.426 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:22.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.537+0000 7ffa5629d640 10 monclient: get_monmap_and_config 2024-03-25T18:23:22.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.537+0000 7ffa5629d640 10 monclient: build_initial_monmap 2024-03-25T18:23:22.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.537+0000 7ffa5629d640 10 monclient: monmap: 2024-03-25T18:23:22.538 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:22.538 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:22.539 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:23:22.538276+0000 2024-03-25T18:23:22.539 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:23:22.538276+0000 2024-03-25T18:23:22.539 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:22.539 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:22.539 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:22.539 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:23:22.539 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:23:22.539 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:22.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.538+0000 7ffa5629d640 1 Processor -- start 2024-03-25T18:23:22.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.538+0000 7ffa5629d640 1 -- start start 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.538+0000 7ffa5629d640 10 monclient: init 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 10 monclient(hunting): picked mon.noname-b con 0x7ffa500705b0 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 10 monclient(hunting): picked mon.noname-c con 0x7ffa501402d0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 10 monclient(hunting): picked mon.noname-a con 0x7ffa50149ae0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa500718c0 con 0x7ffa50149ae0 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa500714c0 con 0x7ffa501402d0 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa50072b50 con 0x7ffa500705b0 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:22.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.539+0000 7ffa5629d640 10 monclient(hunting): authenticate will time out at 8101.812951s 2024-03-25T18:23:22.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.542+0000 7ffa4ffff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7ffa500705b0 0x7ffa5013d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:36828/0 (socket says 172.21.15.162:36828) 2024-03-25T18:23:22.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.542+0000 7ffa4ffff640 1 -- 172.21.15.162:0/155911221 learned_addr learned my addr 172.21.15.162:0/155911221 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:22.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.548+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884227191 0 0) 0x7ffa50072b50 con 0x7ffa500705b0 2024-03-25T18:23:22.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.548+0000 7ffa4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:22.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.548+0000 7ffa4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:22.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.548+0000 7ffa4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:22.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.548+0000 7ffa4effd640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:23:22.549 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.548+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa38003520 con 0x7ffa500705b0 2024-03-25T18:23:22.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1089396793 0 0) 0x7ffa38003520 con 0x7ffa500705b0 2024-03-25T18:23:22.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 >> v1:172.21.15.191:6789/0 conn(0x7ffa501402d0 legacy=0x7ffa50140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 >> v1:172.21.15.162:6789/0 conn(0x7ffa50149ae0 legacy=0x7ffa5014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.558 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 1 monclient: found mon.noname-b 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5013dd10 con 0x7ffa500705b0 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 10 monclient: _finish_auth 0 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffa40001690 con 0x7ffa500705b0 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa4effd640 10 monclient: dump: 2024-03-25T18:23:22.559 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa5629d640 5 monclient: authenticate success, global_id 4637 2024-03-25T18:23:22.560 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.556+0000 7ffa5629d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:22.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.560+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa400017f0 con 0x7ffa500705b0 2024-03-25T18:23:22.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.560+0000 7ffa4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:22.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.560+0000 7ffa4effd640 1 -- 172.21.15.162:0/155911221 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffa40001c40 con 0x7ffa500705b0 2024-03-25T18:23:22.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.560+0000 7ffa4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.560+0000 7ffa4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.560+0000 7ffa4effd640 10 monclient: dump: 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.560+0000 7ffa5629d640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:22.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.560+0000 7ffa5629d640 4 set_mon_vals no callback set 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.561+0000 7ffa5629d640 10 monclient: shutdown 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.561+0000 7ffa5629d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.561+0000 7ffa5629d640 1 -- 172.21.15.162:0/155911221 >> v1:172.21.15.162:6790/0 conn(0x7ffa500705b0 legacy=0x7ffa5013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.561+0000 7ffa5629d640 1 -- 172.21.15.162:0/155911221 shutdown_connections 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.561+0000 7ffa5629d640 1 -- 172.21.15.162:0/155911221 >> 172.21.15.162:0/155911221 conn(0x7ffa500b5d10 msgr2=0x7ffa500b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.562+0000 7ffa5629d640 1 -- 172.21.15.162:0/155911221 shutdown_connections 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.562+0000 7ffa5629d640 1 -- 172.21.15.162:0/155911221 wait complete. 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.562+0000 7ffa5629d640 10 monclient: build_initial_monmap 2024-03-25T18:23:22.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.562+0000 7ffa5629d640 10 monclient: monmap: 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.563+0000 7ffa5629d640 1 librados: starting msgr at 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.563+0000 7ffa5629d640 1 librados: starting objecter 2024-03-25T18:23:22.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.563+0000 7ffa5629d640 1 Processor -- start 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.563+0000 7ffa5629d640 1 -- start start 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.563+0000 7ffa5629d640 1 librados: setting wanted keys 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.563+0000 7ffa5629d640 1 librados: calling monclient init 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.563+0000 7ffa5629d640 10 monclient: init 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 10 monclient(hunting): picked mon.noname-a con 0x7ffa500705b0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 10 monclient(hunting): picked mon.noname-c con 0x7ffa501402d0 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 10 monclient(hunting): picked mon.noname-b con 0x7ffa50149ae0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa501d5590 con 0x7ffa500705b0 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5013dfd0 con 0x7ffa50149ae0 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5013fcd0 con 0x7ffa501402d0 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:22.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.564+0000 7ffa5629d640 10 monclient(hunting): authenticate will time out at 8101.838204s 2024-03-25T18:23:22.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.581+0000 7ffa5483a640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7ffa50149ae0 0x7ffa501dfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:45804/0 (socket says 172.21.15.162:45804) 2024-03-25T18:23:22.582 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.581+0000 7ffa5483a640 1 -- 172.21.15.162:0/919538988 learned_addr learned my addr 172.21.15.162:0/919538988 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:22.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.583+0000 7ffa4cff9640 10 client.?.objecter ms_handle_connect 0x7ffa50149ae0 2024-03-25T18:23:22.584 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.583+0000 7ffa4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:22.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.589+0000 7ffa4cff9640 10 client.?.objecter ms_handle_connect 0x7ffa501402d0 2024-03-25T18:23:22.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.589+0000 7ffa4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:22.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.597+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490384887 0 0) 0x7ffa5013fcd0 con 0x7ffa501402d0 2024-03-25T18:23:22.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.597+0000 7ffa4cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:22.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.597+0000 7ffa4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:22.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.597+0000 7ffa4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:22.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.597+0000 7ffa4cff9640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:23:22.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.597+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa3c004eb0 con 0x7ffa501402d0 2024-03-25T18:23:22.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.597+0000 7ffa4cff9640 10 client.?.objecter ms_handle_connect 0x7ffa500705b0 2024-03-25T18:23:22.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.597+0000 7ffa4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:22.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.608+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488559737 0 0) 0x7ffa501d5590 con 0x7ffa500705b0 2024-03-25T18:23:22.609 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.608+0000 7ffa4cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:22.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.608+0000 7ffa4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:22.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.608+0000 7ffa4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:22.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.608+0000 7ffa4cff9640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:23:22.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.608+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5013fcd0 con 0x7ffa500705b0 2024-03-25T18:23:22.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.617+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1129945630 0 0) 0x7ffa3c004eb0 con 0x7ffa501402d0 2024-03-25T18:23:22.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.618+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa501d5590 con 0x7ffa501402d0 2024-03-25T18:23:22.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.618+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffa44001930 con 0x7ffa501402d0 2024-03-25T18:23:22.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.618+0000 7ffa4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.618+0000 7ffa4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:22.619 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.618+0000 7ffa4cff9640 10 monclient(hunting): dump: 2024-03-25T18:23:22.619 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:22.619 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:22.620 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.618+0000 7ffa4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:22.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.623+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1473205080 0 0) 0x7ffa5013fcd0 con 0x7ffa500705b0 2024-03-25T18:23:22.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.624+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa3c004eb0 con 0x7ffa500705b0 2024-03-25T18:23:22.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.624+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffa40001b00 con 0x7ffa500705b0 2024-03-25T18:23:22.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.624+0000 7ffa4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:22.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.624+0000 7ffa4cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:22.625 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.624+0000 7ffa4cff9640 10 monclient(hunting): dump: 2024-03-25T18:23:22.625 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:22.625 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:22.626 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:22.626 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:22.626 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:22.626 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:22.626 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:22.626 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:22.626 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:22.626 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:22.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2609340040 0 0) 0x7ffa3c004eb0 con 0x7ffa500705b0 2024-03-25T18:23:22.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 >> v1:172.21.15.162:6790/0 conn(0x7ffa501402d0 legacy=0x7ffa501d5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.688 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 >> v1:172.21.15.191:6789/0 conn(0x7ffa50149ae0 legacy=0x7ffa501dfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 1 monclient: found mon.a 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa501d6160 con 0x7ffa500705b0 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 10 monclient: _renew_subs 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:22.689 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa50136030 con 0x7ffa500705b0 2024-03-25T18:23:22.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:22.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 10 monclient: _renew_subs 2024-03-25T18:23:22.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:22.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5013eb00 con 0x7ffa500705b0 2024-03-25T18:23:22.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.687+0000 7ffa5629d640 1 librados: init done 2024-03-25T18:23:22.690 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.688+0000 7ffa5629d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:22.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.688+0000 7ffa5629d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:22.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.688+0000 7ffa5629d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:22.691 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:22.688+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa500627b0 con 0x7ffa500705b0 2024-03-25T18:23:23.013 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.012+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa400011e0 con 0x7ffa500705b0 2024-03-25T18:23:23.013 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.012+0000 7ffa4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:23.013 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.012+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ffa40004bf0 con 0x7ffa500705b0 2024-03-25T18:23:23.013 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.012+0000 7ffa4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:23.013 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.012+0000 7ffa4cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:23.013 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.012+0000 7ffa4cff9640 10 monclient: dump: 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:23.014 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:23.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.012+0000 7ffa4e7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:23.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.013+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7ffa4001abc0 con 0x7ffa500705b0 2024-03-25T18:23:23.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.013+0000 7ffa4f7fe640 10 monclient: get_auth_request con 0x7ffa3c06de80 auth_method 0 2024-03-25T18:23:23.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.014+0000 7ffa4cff9640 10 client.4659.objecter ms_handle_connect 0x7ffa3c06de80 2024-03-25T18:23:23.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.017+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7ffa400547d0 con 0x7ffa500705b0 2024-03-25T18:23:23.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.017+0000 7ffa4cff9640 10 client.4659.objecter ms_dispatch 0x7ffa50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:23.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.017+0000 7ffa4cff9640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:23.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.017+0000 7ffa4cff9640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:23.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.017+0000 7ffa4cff9640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T18:23:23.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.017+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa400502e0 con 0x7ffa500705b0 2024-03-25T18:23:23.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.017+0000 7ffa4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:23.019 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.017+0000 7ffa4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:23.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.276+0000 7ffa5629d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:23.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.276+0000 7ffa5629d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:23.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.276+0000 7ffa5629d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:23.278 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.276+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa501d62f0 con 0x7ffa500705b0 2024-03-25T18:23:23.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.298+0000 7ffa4cff9640 1 -- 172.21.15.162:0/919538988 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7ffa40052210 con 0x7ffa500705b0 2024-03-25T18:23:23.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.298+0000 7ffa4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:23.299 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.298+0000 7ffa4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:23.300 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:23:23.300 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:23.302 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.300+0000 7ffa5629d640 10 librados: watch_flush enter 2024-03-25T18:23:23.302 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.300+0000 7ffa5629d640 10 librados: watch_flush exit 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.300+0000 7ffa5629d640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.300+0000 7ffa5629d640 10 client.4659.objecter successfully canceled tick 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.300+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 >> v1:172.21.15.191:6800/1552703795 conn(0x7ffa3c06de80 legacy=0x7ffa3c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.300+0000 7ffa5629d640 10 monclient: shutdown 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.300+0000 7ffa5629d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.300+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 >> v1:172.21.15.162:6789/0 conn(0x7ffa500705b0 legacy=0x7ffa5013f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.301+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 shutdown_connections 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.301+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 >> 172.21.15.162:0/919538988 conn(0x7ffa500b5d10 msgr2=0x7ffa50140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.301+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 shutdown_connections 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.301+0000 7ffa5629d640 1 -- 172.21.15.162:0/919538988 wait complete. 2024-03-25T18:23:23.303 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:23.301+0000 7ffa5629d640 1 librados: shutdown 2024-03-25T18:23:23.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:30.121 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.233+0000 7f68a817a640 10 monclient: get_monmap_and_config 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.233+0000 7f68a817a640 10 monclient: build_initial_monmap 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.233+0000 7f68a817a640 10 monclient: monmap: 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:23:30.234829+0000 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:23:30.234829+0000 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:23:30.235 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:23:30.236 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:30.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.234+0000 7f68a817a640 1 Processor -- start 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.234+0000 7f68a817a640 1 -- start start 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.234+0000 7f68a817a640 10 monclient: init 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 10 monclient(hunting): picked mon.noname-a con 0x7f68a00a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 10 monclient(hunting): picked mon.noname-b con 0x7f68a013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 10 monclient(hunting): picked mon.noname-c con 0x7f68a01430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0142520 con 0x7f68a00a98f0 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a00b28a0 con 0x7f68a01430c0 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a00b2a30 con 0x7f68a013f710 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:30.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.235+0000 7f68a817a640 10 monclient(hunting): authenticate will time out at 8109.508961s 2024-03-25T18:23:30.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.242+0000 7f68a5f16640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f68a00a98f0 0x7f68a00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:54934/0 (socket says 172.21.15.162:54934) 2024-03-25T18:23:30.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.242+0000 7f68a5f16640 1 -- 172.21.15.162:0/563907688 learned_addr learned my addr 172.21.15.162:0/563907688 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:30.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.258+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925853311 0 0) 0x7f68a00b28a0 con 0x7f68a01430c0 2024-03-25T18:23:30.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.258+0000 7f68a4f14640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.258+0000 7f68a4f14640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.258+0000 7f68a4f14640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.258+0000 7f68a4f14640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:23:30.260 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.258+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c003520 con 0x7f68a01430c0 2024-03-25T18:23:30.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.261+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827149671 0 0) 0x7f68a0142520 con 0x7f68a00a98f0 2024-03-25T18:23:30.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.261+0000 7f68a4f14640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.261+0000 7f68a4f14640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.261+0000 7f68a4f14640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.261+0000 7f68a4f14640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:23:30.262 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.261+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a00b28a0 con 0x7f68a00a98f0 2024-03-25T18:23:30.266 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.265+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280744934 0 0) 0x7f68a00b2a30 con 0x7f68a013f710 2024-03-25T18:23:30.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.265+0000 7f68a4f14640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.265+0000 7f68a4f14640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.265+0000 7f68a4f14640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.265+0000 7f68a4f14640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:23:30.267 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.265+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0142520 con 0x7f68a013f710 2024-03-25T18:23:30.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2243575399 0 0) 0x7f688c003520 con 0x7f68a01430c0 2024-03-25T18:23:30.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 >> v1:172.21.15.162:6790/0 conn(0x7f68a013f710 legacy=0x7f68a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 >> v1:172.21.15.162:6789/0 conn(0x7f68a00a98f0 legacy=0x7f68a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 10 monclient: _finish_hunting 0 2024-03-25T18:23:30.269 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 1 monclient: found mon.noname-c 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a00b44c0 con 0x7f68a01430c0 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 10 monclient: _finish_auth 0 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 10 monclient: _check_auth_tickets 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6898001690 con 0x7f68a01430c0 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a4f14640 10 monclient: dump: 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:30.270 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:30.271 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.271 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:30.271 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:30.271 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:30.271 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:30.271 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:30.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a817a640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:23:30.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.268+0000 7f68a817a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68980017f0 con 0x7f68a01430c0 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a4f14640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a4f14640 1 -- 172.21.15.162:0/563907688 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6898001c00 con 0x7f68a01430c0 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a4f14640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a4f14640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a4f14640 10 monclient: dump: 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:30.283 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a817a640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:30.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a817a640 4 set_mon_vals no callback set 2024-03-25T18:23:30.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a817a640 10 monclient: shutdown 2024-03-25T18:23:30.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a817a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:30.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.282+0000 7f68a817a640 1 -- 172.21.15.162:0/563907688 >> v1:172.21.15.191:6789/0 conn(0x7f68a01430c0 legacy=0x7f68a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.283+0000 7f68a817a640 1 -- 172.21.15.162:0/563907688 shutdown_connections 2024-03-25T18:23:30.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.283+0000 7f68a817a640 1 -- 172.21.15.162:0/563907688 >> 172.21.15.162:0/563907688 conn(0x7f68a00b7ea0 msgr2=0x7f68a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:30.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.283+0000 7f68a817a640 1 -- 172.21.15.162:0/563907688 shutdown_connections 2024-03-25T18:23:30.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.283+0000 7f68a817a640 1 -- 172.21.15.162:0/563907688 wait complete. 2024-03-25T18:23:30.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.284+0000 7f68a817a640 10 monclient: build_initial_monmap 2024-03-25T18:23:30.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.284+0000 7f68a817a640 10 monclient: monmap: 2024-03-25T18:23:30.286 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:30.286 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:30.286 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:23:30.286 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:23:30.286 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.284+0000 7f68a817a640 1 librados: starting msgr at 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.284+0000 7f68a817a640 1 librados: starting objecter 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.284+0000 7f68a817a640 1 Processor -- start 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 1 -- start start 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 1 librados: setting wanted keys 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 1 librados: calling monclient init 2024-03-25T18:23:30.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 10 monclient: init 2024-03-25T18:23:30.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:30.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:30.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 10 monclient(hunting): picked mon.noname-b con 0x7f68a00a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:30.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 10 monclient(hunting): picked mon.noname-c con 0x7f68a013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:30.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 10 monclient(hunting): picked mon.noname-a con 0x7f68a01430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:30.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a01369d0 con 0x7f68a01430c0 2024-03-25T18:23:30.288 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0133a60 con 0x7f68a00a98f0 2024-03-25T18:23:30.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a00ab9e0 con 0x7f68a013f710 2024-03-25T18:23:30.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:30.289 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.285+0000 7f68a817a640 10 monclient(hunting): authenticate will time out at 8109.559470s 2024-03-25T18:23:30.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.296+0000 7f68a5f16640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f68a00a98f0 0x7f68a0132c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:35814/0 (socket says 172.21.15.162:35814) 2024-03-25T18:23:30.297 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.296+0000 7f68a5f16640 1 -- 172.21.15.162:0/1125047249 learned_addr learned my addr 172.21.15.162:0/1125047249 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:30.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.297+0000 7f68967fc640 10 client.?.objecter ms_handle_connect 0x7f68a00a98f0 2024-03-25T18:23:30.298 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.297+0000 7f68967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:30.305 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.304+0000 7f68967fc640 10 client.?.objecter ms_handle_connect 0x7f68a013f710 2024-03-25T18:23:30.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.304+0000 7f68967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:30.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.305+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231215546 0 0) 0x7f68a0133a60 con 0x7f68a00a98f0 2024-03-25T18:23:30.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.305+0000 7f68967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.305+0000 7f68967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.306 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.305+0000 7f68967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.305+0000 7f68967fc640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:23:30.307 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.305+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6884004eb0 con 0x7f68a00a98f0 2024-03-25T18:23:30.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.308+0000 7f68967fc640 10 client.?.objecter ms_handle_connect 0x7f68a01430c0 2024-03-25T18:23:30.309 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.308+0000 7f68967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:30.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.317+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3159642046 0 0) 0x7f6884004eb0 con 0x7f68a00a98f0 2024-03-25T18:23:30.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.317+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a0133a60 con 0x7f68a00a98f0 2024-03-25T18:23:30.318 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.317+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f689c001d70 con 0x7f68a00a98f0 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.317+0000 7f68967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.317+0000 7f68967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.317+0000 7f68967fc640 10 monclient(hunting): dump: 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:30.319 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.317+0000 7f68967fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:30.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.320+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165300982 0 0) 0x7f68a01369d0 con 0x7f68a01430c0 2024-03-25T18:23:30.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.320+0000 7f68967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:30.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.320+0000 7f68967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:30.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.320+0000 7f68967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:30.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.320+0000 7f68967fc640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:23:30.321 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.320+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6884004eb0 con 0x7f68a01430c0 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2770077998 0 0) 0x7f68a0133a60 con 0x7f68a00a98f0 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 >> v1:172.21.15.162:6790/0 conn(0x7f68a013f710 legacy=0x7f68a01362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 >> v1:172.21.15.162:6789/0 conn(0x7f68a01430c0 legacy=0x7f68a0133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 1 monclient: found mon.b 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a00abcc0 con 0x7f68a00a98f0 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:23:30.332 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 10 monclient: _renew_subs 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a00abfa0 con 0x7f68a00a98f0 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 10 monclient: _renew_subs 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a0137e40 con 0x7f68a00a98f0 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.330+0000 7f68a817a640 1 librados: init done 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.331+0000 7f68a817a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.331+0000 7f68a817a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:30.333 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.331+0000 7f68a817a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:30.334 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.331+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a0139060 con 0x7f68a00a98f0 2024-03-25T18:23:30.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.542+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f689c002010 con 0x7f68a00a98f0 2024-03-25T18:23:30.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.542+0000 7f68967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:30.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.542+0000 7f6897fff640 4 set_mon_vals no callback set 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.599+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f689c004fb0 con 0x7f68a00a98f0 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.600+0000 7f68967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.600+0000 7f68967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.600+0000 7f68967fc640 10 monclient: dump: 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:30.601 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:30.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.600+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f689c01af80 con 0x7f68a00a98f0 2024-03-25T18:23:30.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.601+0000 7f68a5715640 10 monclient: get_auth_request con 0x7f688406dd90 auth_method 0 2024-03-25T18:23:30.602 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.601+0000 7f68967fc640 10 client.4642.objecter ms_handle_connect 0x7f688406dd90 2024-03-25T18:23:30.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.606+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f689c054c30 con 0x7f68a00a98f0 2024-03-25T18:23:30.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.606+0000 7f68967fc640 10 client.4642.objecter ms_dispatch 0x7f68a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:30.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.606+0000 7f68967fc640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:30.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.606+0000 7f68967fc640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:30.607 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.606+0000 7f68967fc640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T18:23:30.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.609+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f689c050570 con 0x7f68a00a98f0 2024-03-25T18:23:30.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.609+0000 7f68967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:30.610 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.609+0000 7f68967fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:30.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.865+0000 7f68a817a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:30.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.865+0000 7f68a817a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:30.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.865+0000 7f68a817a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:30.866 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.865+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68a00627b0 con 0x7f68a00a98f0 2024-03-25T18:23:30.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.893+0000 7f68967fc640 1 -- 172.21.15.162:0/1125047249 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f689c052670 con 0x7f68a00a98f0 2024-03-25T18:23:30.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.893+0000 7f68967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:30.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.893+0000 7f68967fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:30.895 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:23:30.895 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:30.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 10 librados: watch_flush enter 2024-03-25T18:23:30.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 10 librados: watch_flush exit 2024-03-25T18:23:30.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T18:23:30.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 10 client.4642.objecter successfully canceled tick 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 >> v1:172.21.15.191:6800/1552703795 conn(0x7f688406dd90 legacy=0x7f6884070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 10 monclient: shutdown 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 >> v1:172.21.15.191:6789/0 conn(0x7f68a00a98f0 legacy=0x7f68a0132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 shutdown_connections 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 >> 172.21.15.162:0/1125047249 conn(0x7f68a00b7ea0 msgr2=0x7f68a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 shutdown_connections 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 1 -- 172.21.15.162:0/1125047249 wait complete. 2024-03-25T18:23:30.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:30.896+0000 7f68a817a640 1 librados: shutdown 2024-03-25T18:23:30.910 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:37.716 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.829+0000 7f1b27925640 10 monclient: get_monmap_and_config 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.829+0000 7f1b27925640 10 monclient: build_initial_monmap 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.829+0000 7f1b27925640 10 monclient: monmap: 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:23:37.830231+0000 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:23:37.830231+0000 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:37.830 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:23:37.831 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:23:37.831 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.829+0000 7f1b27925640 1 Processor -- start 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 1 -- start start 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 10 monclient: init 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 10 monclient(hunting): picked mon.noname-b con 0x7f1b200a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 10 monclient(hunting): picked mon.noname-a con 0x7f1b2013f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 10 monclient(hunting): picked mon.noname-c con 0x7f1b201430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b20142520 con 0x7f1b2013f710 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.830+0000 7f1b27925640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200b28a0 con 0x7f1b201430c0 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.831+0000 7f1b27925640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200b2a30 con 0x7f1b200a98f0 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.831+0000 7f1b27925640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:37.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.831+0000 7f1b27925640 10 monclient(hunting): authenticate will time out at 8117.104683s 2024-03-25T18:23:37.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.837+0000 7f1b24ec0640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f1b2013f710 0x7f1b201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:50474/0 (socket says 172.21.15.162:50474) 2024-03-25T18:23:37.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.837+0000 7f1b24ec0640 1 -- 172.21.15.162:0/2208488767 learned_addr learned my addr 172.21.15.162:0/2208488767 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:37.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.846+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049314083 0 0) 0x7f1b200b28a0 con 0x7f1b201430c0 2024-03-25T18:23:37.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.847+0000 7f1b17fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:37.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.847+0000 7f1b17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:37.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.847+0000 7f1b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:37.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.847+0000 7f1b17fff640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:23:37.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.847+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b0c003520 con 0x7f1b201430c0 2024-03-25T18:23:37.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580049200 0 0) 0x7f1b0c003520 con 0x7f1b201430c0 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 >> v1:172.21.15.162:6790/0 conn(0x7f1b200a98f0 legacy=0x7f1b200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 >> v1:172.21.15.162:6789/0 conn(0x7f1b2013f710 legacy=0x7f1b201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 1 monclient: found mon.noname-c 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b200b44c0 con 0x7f1b201430c0 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 10 monclient: _finish_auth 0 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1b18001690 con 0x7f1b201430c0 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b27925640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b27925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:37.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b17fff640 10 monclient: dump: 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:37.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.857+0000 7f1b27925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:37.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.867+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b180017f0 con 0x7f1b201430c0 2024-03-25T18:23:37.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.867+0000 7f1b17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:37.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.867+0000 7f1b17fff640 1 -- 172.21.15.162:0/2208488767 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1b18001c00 con 0x7f1b201430c0 2024-03-25T18:23:37.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.867+0000 7f1b17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:37.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.867+0000 7f1b17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.867+0000 7f1b17fff640 10 monclient: dump: 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.867+0000 7f1b27925640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:37.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.867+0000 7f1b27925640 4 set_mon_vals no callback set 2024-03-25T18:23:37.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.868+0000 7f1b27925640 10 monclient: shutdown 2024-03-25T18:23:37.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.868+0000 7f1b27925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:37.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.868+0000 7f1b27925640 1 -- 172.21.15.162:0/2208488767 >> v1:172.21.15.191:6789/0 conn(0x7f1b201430c0 legacy=0x7f1b2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.868+0000 7f1b27925640 1 -- 172.21.15.162:0/2208488767 shutdown_connections 2024-03-25T18:23:37.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.868+0000 7f1b27925640 1 -- 172.21.15.162:0/2208488767 >> 172.21.15.162:0/2208488767 conn(0x7f1b200b7ea0 msgr2=0x7f1b200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:37.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.868+0000 7f1b27925640 1 -- 172.21.15.162:0/2208488767 shutdown_connections 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.868+0000 7f1b27925640 1 -- 172.21.15.162:0/2208488767 wait complete. 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.868+0000 7f1b27925640 10 monclient: build_initial_monmap 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 10 monclient: monmap: 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 1 librados: starting msgr at 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 1 librados: starting objecter 2024-03-25T18:23:37.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 1 Processor -- start 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 1 -- start start 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 1 librados: setting wanted keys 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 1 librados: calling monclient init 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 10 monclient: init 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.869+0000 7f1b27925640 10 monclient(hunting): picked mon.noname-a con 0x7f1b200a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.870+0000 7f1b27925640 10 monclient(hunting): picked mon.noname-c con 0x7f1b2013f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.870+0000 7f1b27925640 10 monclient(hunting): picked mon.noname-b con 0x7f1b201430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.870+0000 7f1b27925640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b20137740 con 0x7f1b200a98f0 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.870+0000 7f1b27925640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b20136c50 con 0x7f1b201430c0 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.870+0000 7f1b27925640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b20138fc0 con 0x7f1b2013f710 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.870+0000 7f1b27925640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:37.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.870+0000 7f1b27925640 10 monclient(hunting): authenticate will time out at 8117.143807s 2024-03-25T18:23:37.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.875+0000 7f1b256c1640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f1b200a98f0 0x7f1b20137030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:50520/0 (socket says 172.21.15.162:50520) 2024-03-25T18:23:37.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.875+0000 7f1b256c1640 1 -- 172.21.15.162:0/3811079438 learned_addr learned my addr 172.21.15.162:0/3811079438 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:37.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.875+0000 7f1b15ffb640 10 client.?.objecter ms_handle_connect 0x7f1b200a98f0 2024-03-25T18:23:37.876 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.875+0000 7f1b15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:37.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.877+0000 7f1b15ffb640 10 client.?.objecter ms_handle_connect 0x7f1b201430c0 2024-03-25T18:23:37.879 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.877+0000 7f1b15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:37.882 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.881+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168148011 0 0) 0x7f1b20137740 con 0x7f1b200a98f0 2024-03-25T18:23:37.882 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.881+0000 7f1b15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:37.882 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.881+0000 7f1b15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:37.882 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.881+0000 7f1b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:37.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.881+0000 7f1b15ffb640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:23:37.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.881+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04004eb0 con 0x7f1b200a98f0 2024-03-25T18:23:37.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.888+0000 7f1b15ffb640 10 client.?.objecter ms_handle_connect 0x7f1b2013f710 2024-03-25T18:23:37.889 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.888+0000 7f1b15ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:37.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.893+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 97958975 0 0) 0x7f1b04004eb0 con 0x7f1b200a98f0 2024-03-25T18:23:37.894 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.893+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b20137740 con 0x7f1b200a98f0 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.893+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1b1c001db0 con 0x7f1b200a98f0 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.893+0000 7f1b15ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.893+0000 7f1b15ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.893+0000 7f1b15ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:37.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.893+0000 7f1b15ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:37.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.903+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225841791 0 0) 0x7f1b20136c50 con 0x7f1b201430c0 2024-03-25T18:23:37.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.903+0000 7f1b15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:37.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.903+0000 7f1b15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:37.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.903+0000 7f1b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:37.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.903+0000 7f1b15ffb640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:23:37.904 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.903+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b04004eb0 con 0x7f1b201430c0 2024-03-25T18:23:37.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.908+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058965265 0 0) 0x7f1b20138fc0 con 0x7f1b2013f710 2024-03-25T18:23:37.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.908+0000 7f1b15ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:37.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.908+0000 7f1b15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:37.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.908+0000 7f1b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:37.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.908+0000 7f1b15ffb640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:23:37.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.908+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b20136c50 con 0x7f1b2013f710 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3358516994 0 0) 0x7f1b20137740 con 0x7f1b200a98f0 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 >> v1:172.21.15.162:6790/0 conn(0x7f1b2013f710 legacy=0x7f1b20136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 >> v1:172.21.15.191:6789/0 conn(0x7f1b201430c0 legacy=0x7f1b201e1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 1 monclient: found mon.a 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b20139150 con 0x7f1b200a98f0 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:37.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 5 monclient: authenticate success, global_id 4671 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 10 monclient: _renew_subs 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2013b2e0 con 0x7f1b200a98f0 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 10 client.4671.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 10 monclient: _renew_subs 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b201395e0 con 0x7f1b200a98f0 2024-03-25T18:23:37.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.916+0000 7f1b27925640 1 librados: init done 2024-03-25T18:23:37.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.917+0000 7f1b27925640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:37.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.917+0000 7f1b27925640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:37.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.917+0000 7f1b27925640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:37.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:37.917+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b20136de0 con 0x7f1b200a98f0 2024-03-25T18:23:38.235 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.234+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b1c002030 con 0x7f1b200a98f0 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.234+0000 7f1b15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.234+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1b1c003de0 con 0x7f1b200a98f0 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.234+0000 7f1b15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.234+0000 7f1b15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.234+0000 7f1b15ffb640 10 monclient: dump: 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:38.236 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:38.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.234+0000 7f1b177fe640 4 set_mon_vals no callback set 2024-03-25T18:23:38.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.235+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f1b1c003fc0 con 0x7f1b200a98f0 2024-03-25T18:23:38.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.236+0000 7f1b24ec0640 10 monclient: get_auth_request con 0x7f1b0406de60 auth_method 0 2024-03-25T18:23:38.237 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.236+0000 7f1b15ffb640 10 client.4671.objecter ms_handle_connect 0x7f1b0406de60 2024-03-25T18:23:38.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.242+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f1b1c054bf0 con 0x7f1b200a98f0 2024-03-25T18:23:38.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.242+0000 7f1b15ffb640 10 client.4671.objecter ms_dispatch 0x7f1b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:38.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.242+0000 7f1b15ffb640 3 client.4671.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:38.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.242+0000 7f1b15ffb640 3 client.4671.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:38.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.242+0000 7f1b15ffb640 20 client.4671.objecter dump_active .. 0 homeless 2024-03-25T18:23:38.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.242+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b1c050700 con 0x7f1b200a98f0 2024-03-25T18:23:38.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.242+0000 7f1b15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:38.244 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.242+0000 7f1b15ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:38.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.499+0000 7f1b27925640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:38.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.499+0000 7f1b27925640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:38.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.499+0000 7f1b27925640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:23:38.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.499+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b200627b0 con 0x7f1b200a98f0 2024-03-25T18:23:38.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.522+0000 7f1b15ffb640 1 -- 172.21.15.162:0/3811079438 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f1b1c052630 con 0x7f1b200a98f0 2024-03-25T18:23:38.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.523+0000 7f1b15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:38.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.523+0000 7f1b15ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:38.524 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:23:38.524 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:38.527 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.525+0000 7f1b27925640 10 librados: watch_flush enter 2024-03-25T18:23:38.527 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.525+0000 7f1b27925640 10 librados: watch_flush exit 2024-03-25T18:23:38.527 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.525+0000 7f1b27925640 20 client.4671.objecter shutdown clearing up homeless session... 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.525+0000 7f1b27925640 10 client.4671.objecter successfully canceled tick 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.525+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 >> v1:172.21.15.191:6800/1552703795 conn(0x7f1b0406de60 legacy=0x7f1b04070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.525+0000 7f1b27925640 10 monclient: shutdown 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.525+0000 7f1b27925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.525+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 >> v1:172.21.15.162:6789/0 conn(0x7f1b200a98f0 legacy=0x7f1b20137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.526+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 shutdown_connections 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.526+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 >> 172.21.15.162:0/3811079438 conn(0x7f1b200b7ea0 msgr2=0x7f1b20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.526+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 shutdown_connections 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.526+0000 7f1b27925640 1 -- 172.21.15.162:0/3811079438 wait complete. 2024-03-25T18:23:38.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:38.526+0000 7f1b27925640 1 librados: shutdown 2024-03-25T18:23:38.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:38.961 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:51.189 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:38.967 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:51.195 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:38.992 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:09:51.219 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:39.015 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:51.230 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:39.018 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:51.234 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:39.021 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:51.248 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:39.027 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:51.255 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:39.046 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:09:51.262 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:39.061 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:09:51.289 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:45.347 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:45.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.458+0000 7f45e9c06640 10 monclient: get_monmap_and_config 2024-03-25T18:23:45.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.458+0000 7f45e9c06640 10 monclient: build_initial_monmap 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.458+0000 7f45e9c06640 10 monclient: monmap: 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:23:45.459478+0000 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:23:45.459478+0000 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:23:45.460 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:45.461 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.459+0000 7f45e9c06640 1 Processor -- start 2024-03-25T18:23:45.461 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.459+0000 7f45e9c06640 1 -- start start 2024-03-25T18:23:45.461 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.459+0000 7f45e9c06640 10 monclient: init 2024-03-25T18:23:45.461 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 10 monclient(hunting): picked mon.noname-b con 0x7f45e40a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 10 monclient(hunting): picked mon.noname-a con 0x7f45e413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 10 monclient(hunting): picked mon.noname-c con 0x7f45e41430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e4142520 con 0x7f45e413f710 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e40b28a0 con 0x7f45e41430c0 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e40b2a30 con 0x7f45e40a98f0 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:45.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.460+0000 7f45e9c06640 10 monclient(hunting): authenticate will time out at 8124.734119s 2024-03-25T18:23:45.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.475+0000 7f45e3fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f45e41430c0 0x7f45e414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:47342/0 (socket says 172.21.15.162:47342) 2024-03-25T18:23:45.476 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.475+0000 7f45e3fff640 1 -- 172.21.15.162:0/1682260925 learned_addr learned my addr 172.21.15.162:0/1682260925 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:45.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.480+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398948421 0 0) 0x7f45e40b28a0 con 0x7f45e41430c0 2024-03-25T18:23:45.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.480+0000 7f45e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.480+0000 7f45e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.480+0000 7f45e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.480+0000 7f45e27fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:23:45.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.480+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45cc003520 con 0x7f45e41430c0 2024-03-25T18:23:45.487 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.486+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800774526 0 0) 0x7f45e40b2a30 con 0x7f45e40a98f0 2024-03-25T18:23:45.487 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.486+0000 7f45e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.487 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.486+0000 7f45e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.487 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.486+0000 7f45e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.487 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.486+0000 7f45e27fc640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:23:45.487 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.486+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e40b28a0 con 0x7f45e40a98f0 2024-03-25T18:23:45.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.492+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293406275 0 0) 0x7f45e4142520 con 0x7f45e413f710 2024-03-25T18:23:45.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.492+0000 7f45e27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.492+0000 7f45e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.493 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.492+0000 7f45e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.492+0000 7f45e27fc640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:23:45.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.492+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e40b2a30 con 0x7f45e413f710 2024-03-25T18:23:45.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447195977 0 0) 0x7f45cc003520 con 0x7f45e41430c0 2024-03-25T18:23:45.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 >> v1:172.21.15.162:6790/0 conn(0x7f45e40a98f0 legacy=0x7f45e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 >> v1:172.21.15.162:6789/0 conn(0x7f45e413f710 legacy=0x7f45e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:45.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 1 monclient: found mon.noname-c 2024-03-25T18:23:45.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:45.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e40b44c0 con 0x7f45e41430c0 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f45dc001a10 con 0x7f45e41430c0 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e27fc640 10 monclient: dump: 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:45.501 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:45.502 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:45.502 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:45.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e9c06640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:23:45.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.499+0000 7f45e9c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.514+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45dc001b70 con 0x7f45e41430c0 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e27fc640 1 -- 172.21.15.162:0/1682260925 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f45dc001f80 con 0x7f45e41430c0 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e27fc640 10 monclient: dump: 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:45.516 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:45.517 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:45.517 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:45.517 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:45.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:45.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 4 set_mon_vals no callback set 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 10 monclient: shutdown 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 1 -- 172.21.15.162:0/1682260925 >> v1:172.21.15.191:6789/0 conn(0x7f45e41430c0 legacy=0x7f45e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 1 -- 172.21.15.162:0/1682260925 shutdown_connections 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 1 -- 172.21.15.162:0/1682260925 >> 172.21.15.162:0/1682260925 conn(0x7f45e40b7ea0 msgr2=0x7f45e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 1 -- 172.21.15.162:0/1682260925 shutdown_connections 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.515+0000 7f45e9c06640 1 -- 172.21.15.162:0/1682260925 wait complete. 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.516+0000 7f45e9c06640 10 monclient: build_initial_monmap 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.516+0000 7f45e9c06640 10 monclient: monmap: 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:45.519 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.516+0000 7f45e9c06640 1 librados: starting msgr at 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.516+0000 7f45e9c06640 1 librados: starting objecter 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.516+0000 7f45e9c06640 1 Processor -- start 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 1 -- start start 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 1 librados: setting wanted keys 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 1 librados: calling monclient init 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 10 monclient: init 2024-03-25T18:23:45.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 10 monclient(hunting): picked mon.noname-b con 0x7f45e40a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 10 monclient(hunting): picked mon.noname-c con 0x7f45e413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 10 monclient(hunting): picked mon.noname-a con 0x7f45e41430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e4135230 con 0x7f45e41430c0 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e4135da0 con 0x7f45e40a98f0 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e4135f30 con 0x7f45e413f710 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:45.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.517+0000 7f45e9c06640 10 monclient(hunting): authenticate will time out at 8124.791423s 2024-03-25T18:23:45.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.523+0000 7f45e2ffd640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f45e413f710 0x7f45e41dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:45590/0 (socket says 172.21.15.162:45590) 2024-03-25T18:23:45.524 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.523+0000 7f45e2ffd640 1 -- 172.21.15.162:0/1539977053 learned_addr learned my addr 172.21.15.162:0/1539977053 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:45.527 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.526+0000 7f45cbfff640 10 client.?.objecter ms_handle_connect 0x7f45e413f710 2024-03-25T18:23:45.527 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.526+0000 7f45cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:45.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.529+0000 7f45cbfff640 10 client.?.objecter ms_handle_connect 0x7f45e41430c0 2024-03-25T18:23:45.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.529+0000 7f45cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:45.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.531+0000 7f45cbfff640 10 client.?.objecter ms_handle_connect 0x7f45e40a98f0 2024-03-25T18:23:45.532 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.531+0000 7f45cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:45.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.539+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129262530 0 0) 0x7f45e4135f30 con 0x7f45e413f710 2024-03-25T18:23:45.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.539+0000 7f45cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.539+0000 7f45cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.539+0000 7f45cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.539+0000 7f45cbfff640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:23:45.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.539+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45d0004eb0 con 0x7f45e413f710 2024-03-25T18:23:45.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.541+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830965725 0 0) 0x7f45e4135da0 con 0x7f45e40a98f0 2024-03-25T18:23:45.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.541+0000 7f45cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.541+0000 7f45cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.541+0000 7f45cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.541+0000 7f45cbfff640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:23:45.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.541+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e4135f30 con 0x7f45e40a98f0 2024-03-25T18:23:45.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.545+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252978819 0 0) 0x7f45e4135230 con 0x7f45e41430c0 2024-03-25T18:23:45.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.545+0000 7f45cbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:45.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.545+0000 7f45cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:45.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.545+0000 7f45cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:45.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.545+0000 7f45cbfff640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:23:45.546 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.545+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45e4135da0 con 0x7f45e41430c0 2024-03-25T18:23:45.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.562+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 467493549 0 0) 0x7f45d0004eb0 con 0x7f45e413f710 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.562+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45e4135230 con 0x7f45e413f710 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.562+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f45d8001a10 con 0x7f45e413f710 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.562+0000 7f45cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.562+0000 7f45cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.562+0000 7f45cbfff640 10 monclient(hunting): dump: 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:45.564 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:45.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.562+0000 7f45cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:45.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.564+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2586177809 0 0) 0x7f45e4135f30 con 0x7f45e40a98f0 2024-03-25T18:23:45.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.564+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45d0004eb0 con 0x7f45e40a98f0 2024-03-25T18:23:45.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.564+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f45d4001970 con 0x7f45e40a98f0 2024-03-25T18:23:45.565 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.564+0000 7f45cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.564+0000 7f45cbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.564+0000 7f45cbfff640 10 monclient(hunting): dump: 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:45.566 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:45.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.566+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4279738743 0 0) 0x7f45e4135da0 con 0x7f45e41430c0 2024-03-25T18:23:45.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.566+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45e4135f30 con 0x7f45e41430c0 2024-03-25T18:23:45.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.566+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f45dc001e80 con 0x7f45e41430c0 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.566+0000 7f45cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.566+0000 7f45cbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.566+0000 7f45cbfff640 10 monclient(hunting): dump: 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:45.568 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:45.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2760767588 0 0) 0x7f45d0004eb0 con 0x7f45e40a98f0 2024-03-25T18:23:45.588 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 >> v1:172.21.15.162:6790/0 conn(0x7f45e413f710 legacy=0x7f45e41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 >> v1:172.21.15.162:6789/0 conn(0x7f45e41430c0 legacy=0x7f45e41db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 1 monclient: found mon.b 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e41360c0 con 0x7f45e40a98f0 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 10 monclient: _finish_auth 0 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 10 monclient: _renew_subs 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e41d5000 con 0x7f45e40a98f0 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:45.589 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 10 monclient: _renew_subs 2024-03-25T18:23:45.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:45.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e41d41e0 con 0x7f45e40a98f0 2024-03-25T18:23:45.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.587+0000 7f45e9c06640 1 librados: init done 2024-03-25T18:23:45.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.588+0000 7f45e9c06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:45.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.588+0000 7f45e9c06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:45.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.588+0000 7f45e9c06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:45.590 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.588+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e4135090 con 0x7f45e40a98f0 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.863+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45d40035f0 con 0x7f45e40a98f0 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.863+0000 7f45cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.863+0000 7f45e1ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.863+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f45d4003760 con 0x7f45e40a98f0 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.863+0000 7f45cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.863+0000 7f45cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.863+0000 7f45cbfff640 10 monclient: dump: 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:45.864 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:45.865 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:45.865 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:45.865 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:45.865 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:45.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.864+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f45d4019730 con 0x7f45e40a98f0 2024-03-25T18:23:45.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.865+0000 7f45e2ffd640 10 monclient: get_auth_request con 0x7f45d006e110 auth_method 0 2024-03-25T18:23:45.865 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.865+0000 7f45cbfff640 10 client.4654.objecter ms_handle_connect 0x7f45d006e110 2024-03-25T18:23:45.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.869+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f45d4053490 con 0x7f45e40a98f0 2024-03-25T18:23:45.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.869+0000 7f45cbfff640 10 client.4654.objecter ms_dispatch 0x7f45e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:45.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.869+0000 7f45cbfff640 3 client.4654.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:45.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.869+0000 7f45cbfff640 3 client.4654.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:45.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.869+0000 7f45cbfff640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T18:23:45.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.869+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45d404edd0 con 0x7f45e40a98f0 2024-03-25T18:23:45.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.869+0000 7f45cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:45.870 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:45.869+0000 7f45cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:46.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.125+0000 7f45e9c06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:46.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.125+0000 7f45e9c06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:46.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.125+0000 7f45e9c06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:46.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.125+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45e413b700 con 0x7f45e40a98f0 2024-03-25T18:23:46.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.152+0000 7f45cbfff640 1 -- 172.21.15.162:0/1539977053 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f45d4050ed0 con 0x7f45e40a98f0 2024-03-25T18:23:46.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.152+0000 7f45cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:46.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.152+0000 7f45cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:46.153 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:23:46.153 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:46.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.154+0000 7f45e9c06640 10 librados: watch_flush enter 2024-03-25T18:23:46.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.154+0000 7f45e9c06640 10 librados: watch_flush exit 2024-03-25T18:23:46.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.154+0000 7f45e9c06640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T18:23:46.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.154+0000 7f45e9c06640 10 client.4654.objecter successfully canceled tick 2024-03-25T18:23:46.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.154+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 >> v1:172.21.15.191:6800/1552703795 conn(0x7f45d006e110 legacy=0x7f45d00705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:46.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.154+0000 7f45e9c06640 10 monclient: shutdown 2024-03-25T18:23:46.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.154+0000 7f45e9c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:46.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.154+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 >> v1:172.21.15.191:6789/0 conn(0x7f45e40a98f0 legacy=0x7f45e41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:46.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.155+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 shutdown_connections 2024-03-25T18:23:46.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.155+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 >> 172.21.15.162:0/1539977053 conn(0x7f45e40b7ea0 msgr2=0x7f45e414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:46.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.155+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 shutdown_connections 2024-03-25T18:23:46.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.155+0000 7f45e9c06640 1 -- 172.21.15.162:0/1539977053 wait complete. 2024-03-25T18:23:46.156 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:46.155+0000 7f45e9c06640 1 librados: shutdown 2024-03-25T18:23:46.167 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:52.974 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:53.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.088+0000 7fa0057d0640 10 monclient: get_monmap_and_config 2024-03-25T18:23:53.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.088+0000 7fa0057d0640 10 monclient: build_initial_monmap 2024-03-25T18:23:53.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.088+0000 7fa0057d0640 10 monclient: monmap: 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:23:53.089420+0000 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:23:53.089420+0000 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:23:53.090 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:53.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.089+0000 7fa0057d0640 1 Processor -- start 2024-03-25T18:23:53.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.089+0000 7fa0057d0640 1 -- start start 2024-03-25T18:23:53.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.089+0000 7fa0057d0640 10 monclient: init 2024-03-25T18:23:53.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:53.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:53.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa0000a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:53.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa00013f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:53.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa0001430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:53.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa000142520 con 0x7fa0001430c0 2024-03-25T18:23:53.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000b28a0 con 0x7fa00013f710 2024-03-25T18:23:53.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000b2a30 con 0x7fa0000a98f0 2024-03-25T18:23:53.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:53.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.090+0000 7fa0057d0640 10 monclient(hunting): authenticate will time out at 8132.363891s 2024-03-25T18:23:53.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.105+0000 7f9ff7fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa00013f710 0x7fa0001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:52124/0 (socket says 172.21.15.162:52124) 2024-03-25T18:23:53.106 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.105+0000 7f9ff7fff640 1 -- 172.21.15.162:0/3182057998 learned_addr learned my addr 172.21.15.162:0/3182057998 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:53.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.118+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489516639 0 0) 0x7fa0000b28a0 con 0x7fa00013f710 2024-03-25T18:23:53.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.119+0000 7f9ffe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:53.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.119+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:53.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.119+0000 7f9ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:53.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.119+0000 7f9ffe7fc640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:23:53.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.119+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec003520 con 0x7fa00013f710 2024-03-25T18:23:53.134 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 949404374 0 0) 0x7f9fec003520 con 0x7fa00013f710 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 >> v1:172.21.15.162:6790/0 conn(0x7fa0000a98f0 legacy=0x7fa0000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 >> v1:172.21.15.162:6789/0 conn(0x7fa0001430c0 legacy=0x7fa00014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 1 monclient: found mon.noname-c 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0000b44c0 con 0x7fa00013f710 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9fe8001690 con 0x7fa00013f710 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7f9ffe7fc640 10 monclient: dump: 2024-03-25T18:23:53.135 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7fa0057d0640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:23:53.136 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.133+0000 7fa0057d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:53.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.142+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fe80017f0 con 0x7fa00013f710 2024-03-25T18:23:53.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.142+0000 7f9ffe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:53.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.142+0000 7f9ffe7fc640 1 -- 172.21.15.162:0/3182057998 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9fe8001c00 con 0x7fa00013f710 2024-03-25T18:23:53.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.142+0000 7fa0057d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:53.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.142+0000 7fa0057d0640 4 set_mon_vals no callback set 2024-03-25T18:23:53.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.142+0000 7f9ffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.142+0000 7f9ffe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:53.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.142+0000 7f9ffe7fc640 10 monclient: dump: 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:53.144 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:53.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 10 monclient: shutdown 2024-03-25T18:23:53.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:53.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 1 -- 172.21.15.162:0/3182057998 >> v1:172.21.15.191:6789/0 conn(0x7fa00013f710 legacy=0x7fa0001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 1 -- 172.21.15.162:0/3182057998 shutdown_connections 2024-03-25T18:23:53.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 1 -- 172.21.15.162:0/3182057998 >> 172.21.15.162:0/3182057998 conn(0x7fa0000b7ea0 msgr2=0x7fa0000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:53.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 1 -- 172.21.15.162:0/3182057998 shutdown_connections 2024-03-25T18:23:53.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 1 -- 172.21.15.162:0/3182057998 wait complete. 2024-03-25T18:23:53.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 10 monclient: build_initial_monmap 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.143+0000 7fa0057d0640 10 monclient: monmap: 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.144+0000 7fa0057d0640 1 librados: starting msgr at 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.144+0000 7fa0057d0640 1 librados: starting objecter 2024-03-25T18:23:53.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.144+0000 7fa0057d0640 1 Processor -- start 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.144+0000 7fa0057d0640 1 -- start start 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.144+0000 7fa0057d0640 1 librados: setting wanted keys 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.144+0000 7fa0057d0640 1 librados: calling monclient init 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.144+0000 7fa0057d0640 10 monclient: init 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 10 monclient(hunting): picked mon.noname-b con 0x7fa0000a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 10 monclient(hunting): picked mon.noname-a con 0x7fa00013f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 10 monclient(hunting): picked mon.noname-c con 0x7fa0001430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa00013be70 con 0x7fa00013f710 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa000131660 con 0x7fa0000a98f0 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0001317f0 con 0x7fa0001430c0 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:53.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.145+0000 7fa0057d0640 10 monclient(hunting): authenticate will time out at 8132.418821s 2024-03-25T18:23:53.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.154+0000 7f9ffeffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fa0000a98f0 0x7fa000130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:52188/0 (socket says 172.21.15.162:52188) 2024-03-25T18:23:53.155 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.154+0000 7f9ffeffd640 1 -- 172.21.15.162:0/1192404110 learned_addr learned my addr 172.21.15.162:0/1192404110 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:23:53.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.157+0000 7f9ff77fe640 10 client.?.objecter ms_handle_connect 0x7fa0001430c0 2024-03-25T18:23:53.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.157+0000 7f9ff77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:53.158 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.157+0000 7f9ff77fe640 10 client.?.objecter ms_handle_connect 0x7fa0000a98f0 2024-03-25T18:23:53.159 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.157+0000 7f9ff77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:53.166 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.165+0000 7f9ff77fe640 10 client.?.objecter ms_handle_connect 0x7fa00013f710 2024-03-25T18:23:53.166 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.165+0000 7f9ff77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:53.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.167+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797287171 0 0) 0x7fa000131660 con 0x7fa0000a98f0 2024-03-25T18:23:53.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.167+0000 7f9ff77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:53.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.167+0000 7f9ff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:53.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.167+0000 7f9ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:53.168 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.167+0000 7f9ff77fe640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:23:53.169 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.167+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe0004eb0 con 0x7fa0000a98f0 2024-03-25T18:23:53.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.183+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276171188 0 0) 0x7fa00013be70 con 0x7fa00013f710 2024-03-25T18:23:53.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.183+0000 7f9ff77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:53.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.183+0000 7f9ff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:53.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.183+0000 7f9ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:53.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.183+0000 7f9ff77fe640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:23:53.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.183+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa000131660 con 0x7fa00013f710 2024-03-25T18:23:53.190 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.188+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266526140 0 0) 0x7fa0001317f0 con 0x7fa0001430c0 2024-03-25T18:23:53.190 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.188+0000 7f9ff77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:53.190 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.188+0000 7f9ff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:53.190 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.188+0000 7f9ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:53.190 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.188+0000 7f9ff77fe640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:23:53.190 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.188+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa00013be70 con 0x7fa0001430c0 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.189+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 177180546 0 0) 0x7f9fe0004eb0 con 0x7fa0000a98f0 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.189+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0001317f0 con 0x7fa0000a98f0 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.189+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9ff0001a30 con 0x7fa0000a98f0 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.189+0000 7f9ff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.189+0000 7f9ff77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.189+0000 7f9ff77fe640 10 monclient(hunting): dump: 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:53.191 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:53.192 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.189+0000 7f9ff77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:53.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.206+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188828140 0 0) 0x7fa000131660 con 0x7fa00013f710 2024-03-25T18:23:53.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.206+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fe0004eb0 con 0x7fa00013f710 2024-03-25T18:23:53.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.206+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9fe8001b40 con 0x7fa00013f710 2024-03-25T18:23:53.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.206+0000 7f9ff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.206+0000 7f9ff77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:53.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.206+0000 7f9ff77fe640 10 monclient(hunting): dump: 2024-03-25T18:23:53.207 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:53.207 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:53.208 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.208 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.208 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.208 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:53.208 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:53.208 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:53.208 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:53.208 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:53.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3585595612 0 0) 0x7fa0001317f0 con 0x7fa0000a98f0 2024-03-25T18:23:53.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 >> v1:172.21.15.162:6790/0 conn(0x7fa0001430c0 legacy=0x7fa0001e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 >> v1:172.21.15.162:6789/0 conn(0x7fa00013f710 legacy=0x7fa0001d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 1 monclient: found mon.b 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa000131980 con 0x7fa0000a98f0 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7f9ff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 10 monclient: _renew_subs 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa000135c60 con 0x7fa0000a98f0 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 10 monclient: _renew_subs 2024-03-25T18:23:53.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:53.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0000b7770 con 0x7fa0000a98f0 2024-03-25T18:23:53.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.210+0000 7fa0057d0640 1 librados: init done 2024-03-25T18:23:53.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.211+0000 7fa0057d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:53.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.211+0000 7fa0057d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:53.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.211+0000 7fa0057d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:53.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.211+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0001371d0 con 0x7fa0000a98f0 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.489+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ff0001c70 con 0x7fa0000a98f0 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.489+0000 7f9ff77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.489+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9ff0003a60 con 0x7fa0000a98f0 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.489+0000 7f9ff77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.489+0000 7f9ff77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.489+0000 7f9ff77fe640 10 monclient: dump: 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.490 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.489+0000 7f9ffdffb640 4 set_mon_vals no callback set 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.490+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f9ff0003c40 con 0x7fa0000a98f0 2024-03-25T18:23:53.491 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.490+0000 7f9ff7fff640 10 monclient: get_auth_request con 0x7f9fe006e000 auth_method 0 2024-03-25T18:23:53.492 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.491+0000 7f9ff77fe640 10 client.4660.objecter ms_handle_connect 0x7f9fe006e000 2024-03-25T18:23:53.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.495+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f9ff0054900 con 0x7fa0000a98f0 2024-03-25T18:23:53.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.495+0000 7f9ff77fe640 10 client.4660.objecter ms_dispatch 0x7fa000000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:53.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.495+0000 7f9ff77fe640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:53.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.495+0000 7f9ff77fe640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:53.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.496+0000 7f9ff77fe640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T18:23:53.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.496+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ff0050410 con 0x7fa0000a98f0 2024-03-25T18:23:53.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.496+0000 7f9ff77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:53.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.496+0000 7f9ff77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:53.750 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.749+0000 7fa0057d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:53.751 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.749+0000 7fa0057d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:53.751 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.749+0000 7fa0057d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:23:53.751 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.749+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa000074cc0 con 0x7fa0000a98f0 2024-03-25T18:23:53.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.775+0000 7f9ff77fe640 1 -- 172.21.15.162:0/1192404110 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f9ff0052340 con 0x7fa0000a98f0 2024-03-25T18:23:53.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.775+0000 7f9ff77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:53.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.775+0000 7f9ff77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:53.776 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:23:53.776 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 10 librados: watch_flush enter 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 10 librados: watch_flush exit 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 10 client.4660.objecter successfully canceled tick 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 >> v1:172.21.15.191:6800/1552703795 conn(0x7f9fe006e000 legacy=0x7f9fe00704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 10 monclient: shutdown 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 >> v1:172.21.15.191:6789/0 conn(0x7fa0000a98f0 legacy=0x7fa000130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 shutdown_connections 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 >> 172.21.15.162:0/1192404110 conn(0x7fa0000b7ea0 msgr2=0x7fa000074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 shutdown_connections 2024-03-25T18:23:53.779 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 1 -- 172.21.15.162:0/1192404110 wait complete. 2024-03-25T18:23:53.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:23:53.777+0000 7fa0057d0640 1 librados: shutdown 2024-03-25T18:23:53.789 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:58.872 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:11.099 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:58.875 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:11.103 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:58.917 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:11.133 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:58.918 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:11.134 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:58.949 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:11.165 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:58.962 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:11.190 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:59.000 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:11.228 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:59.011 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:11.238 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:59.031 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:11.258 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:00.595 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:00.707 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.706+0000 7f6829a33640 10 monclient: get_monmap_and_config 2024-03-25T18:24:00.707 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.706+0000 7f6829a33640 10 monclient: build_initial_monmap 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.706+0000 7f6829a33640 10 monclient: monmap: 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:24:00.707726+0000 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:24:00.707726+0000 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:24:00.708 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:00.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.707+0000 7f6829a33640 1 Processor -- start 2024-03-25T18:24:00.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.707+0000 7f6829a33640 1 -- start start 2024-03-25T18:24:00.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.707+0000 7f6829a33640 10 monclient: init 2024-03-25T18:24:00.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:00.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:00.709 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 10 monclient(hunting): picked mon.noname-c con 0x7f68240a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 10 monclient(hunting): picked mon.noname-a con 0x7f682413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 10 monclient(hunting): picked mon.noname-b con 0x7f68241430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6824142520 con 0x7f682413f710 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68240b28a0 con 0x7f68240a98f0 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68240b2a30 con 0x7f68241430c0 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.708+0000 7f6829a33640 10 monclient(hunting): authenticate will time out at 8139.981918s 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.709+0000 7f6822ffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f68240a98f0 0x7f68240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:58690/0 (socket says 172.21.15.162:58690) 2024-03-25T18:24:00.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.709+0000 7f6822ffd640 1 -- 172.21.15.162:0/1946549101 learned_addr learned my addr 172.21.15.162:0/1946549101 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:00.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.800+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901350594 0 0) 0x7f6824142520 con 0x7f682413f710 2024-03-25T18:24:00.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.800+0000 7f6821ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:00.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.800+0000 7f6821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:00.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.800+0000 7f6821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:00.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.800+0000 7f6821ffb640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:24:00.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.800+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f680c003520 con 0x7f682413f710 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3742997754 0 0) 0x7f680c003520 con 0x7f682413f710 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 >> v1:172.21.15.162:6790/0 conn(0x7f68241430c0 legacy=0x7f682414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 >> v1:172.21.15.191:6789/0 conn(0x7f68240a98f0 legacy=0x7f68240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 1 monclient: found mon.noname-a 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68240b44c0 con 0x7f682413f710 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:00.819 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6818001690 con 0x7f682413f710 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6829a33640 5 monclient: authenticate success, global_id 4686 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6829a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6821ffb640 10 monclient: dump: 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:00.820 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:00.821 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:00.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.817+0000 7f6829a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:00.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.826+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68180017f0 con 0x7f682413f710 2024-03-25T18:24:00.827 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.826+0000 7f6821ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.826+0000 7f6821ffb640 1 -- 172.21.15.162:0/1946549101 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6818001c00 con 0x7f682413f710 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.826+0000 7f6829a33640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.826+0000 7f6829a33640 4 set_mon_vals no callback set 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.826+0000 7f6821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.826+0000 7f6821ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.826+0000 7f6821ffb640 10 monclient: dump: 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:00.828 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:00.829 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:00.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.827+0000 7f6829a33640 10 monclient: shutdown 2024-03-25T18:24:00.830 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.827+0000 7f6829a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.827+0000 7f6829a33640 1 -- 172.21.15.162:0/1946549101 >> v1:172.21.15.162:6789/0 conn(0x7f682413f710 legacy=0x7f68241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.827+0000 7f6829a33640 1 -- 172.21.15.162:0/1946549101 shutdown_connections 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.827+0000 7f6829a33640 1 -- 172.21.15.162:0/1946549101 >> 172.21.15.162:0/1946549101 conn(0x7f68240b7ea0 msgr2=0x7f68240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.827+0000 7f6829a33640 1 -- 172.21.15.162:0/1946549101 shutdown_connections 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.827+0000 7f6829a33640 1 -- 172.21.15.162:0/1946549101 wait complete. 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 10 monclient: build_initial_monmap 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 10 monclient: monmap: 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:00.831 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 1 librados: starting msgr at 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 1 librados: starting objecter 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 1 Processor -- start 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 1 -- start start 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 1 librados: setting wanted keys 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 1 librados: calling monclient init 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.828+0000 7f6829a33640 10 monclient: init 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 10 monclient(hunting): picked mon.noname-b con 0x7f68240a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 10 monclient(hunting): picked mon.noname-c con 0x7f682413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 10 monclient(hunting): picked mon.noname-a con 0x7f68241430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:00.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68240abd70 con 0x7f68241430c0 2024-03-25T18:24:00.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68241d7710 con 0x7f68240a98f0 2024-03-25T18:24:00.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68241d8340 con 0x7f682413f710 2024-03-25T18:24:00.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:00.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.829+0000 7f6829a33640 10 monclient(hunting): authenticate will time out at 8140.103118s 2024-03-25T18:24:00.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.835+0000 7f68237fe640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f68241430c0 0x7f68241d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:39184/0 (socket says 172.21.15.162:39184) 2024-03-25T18:24:00.836 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.835+0000 7f68237fe640 1 -- 172.21.15.162:0/1876797534 learned_addr learned my addr 172.21.15.162:0/1876797534 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:00.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.836+0000 7f680bfff640 10 client.?.objecter ms_handle_connect 0x7f68241430c0 2024-03-25T18:24:00.837 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.836+0000 7f680bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:00.840 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.839+0000 7f680bfff640 10 client.?.objecter ms_handle_connect 0x7f68240a98f0 2024-03-25T18:24:00.840 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.839+0000 7f680bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:00.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.845+0000 7f680bfff640 10 client.?.objecter ms_handle_connect 0x7f682413f710 2024-03-25T18:24:00.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.845+0000 7f680bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:00.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.853+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676340127 0 0) 0x7f68240abd70 con 0x7f68241430c0 2024-03-25T18:24:00.854 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.853+0000 7f680bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:00.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.853+0000 7f680bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:00.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.853+0000 7f680bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:00.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.853+0000 7f680bfff640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:24:00.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.853+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6810004eb0 con 0x7f68241430c0 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.866+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1314301689 0 0) 0x7f6810004eb0 con 0x7f68241430c0 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.866+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68240abd70 con 0x7f68241430c0 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.866+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f681c001c30 con 0x7f68241430c0 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.866+0000 7f680bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.866+0000 7f680bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.866+0000 7f680bfff640 10 monclient(hunting): dump: 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:00.868 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:00.869 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:00.869 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:00.869 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:00.869 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.866+0000 7f680bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:00.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1322495679 0 0) 0x7f68240abd70 con 0x7f68241430c0 2024-03-25T18:24:00.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 >> v1:172.21.15.162:6790/0 conn(0x7f682413f710 legacy=0x7f68241d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 >> v1:172.21.15.191:6789/0 conn(0x7f68240a98f0 legacy=0x7f6824137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 1 monclient: found mon.a 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68241d84d0 con 0x7f68241430c0 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 10 monclient: _finish_auth 0 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f680bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 5 monclient: authenticate success, global_id 4689 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 10 monclient: _renew_subs 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68241d41e0 con 0x7f68241430c0 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 10 monclient: _renew_subs 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68241d5150 con 0x7f68241430c0 2024-03-25T18:24:00.884 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.882+0000 7f6829a33640 1 librados: init done 2024-03-25T18:24:00.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.883+0000 7f6829a33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:00.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.883+0000 7f6829a33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:00.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.883+0000 7f6829a33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:00.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:00.883+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68240627b0 con 0x7f68241430c0 2024-03-25T18:24:01.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.238+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f681c003970 con 0x7f68241430c0 2024-03-25T18:24:01.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.238+0000 7f680bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:01.239 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.238+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f681c003d40 con 0x7f68241430c0 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.238+0000 7f680bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.238+0000 7f680bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.238+0000 7f680bfff640 10 monclient: dump: 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:01.240 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.238+0000 7f68217fa640 4 set_mon_vals no callback set 2024-03-25T18:24:01.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.239+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f681c003f60 con 0x7f68241430c0 2024-03-25T18:24:01.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.240+0000 7f6822ffd640 10 monclient: get_auth_request con 0x7f681006dc80 auth_method 0 2024-03-25T18:24:01.241 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.240+0000 7f680bfff640 10 client.4689.objecter ms_handle_connect 0x7f681006dc80 2024-03-25T18:24:01.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.251+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f681c054a10 con 0x7f68241430c0 2024-03-25T18:24:01.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.251+0000 7f680bfff640 10 client.4689.objecter ms_dispatch 0x7f6824000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:01.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.251+0000 7f680bfff640 3 client.4689.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:01.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.251+0000 7f680bfff640 3 client.4689.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:01.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.251+0000 7f680bfff640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T18:24:01.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.251+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f681c050520 con 0x7f68241430c0 2024-03-25T18:24:01.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.251+0000 7f680bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:01.253 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.251+0000 7f680bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:01.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.508+0000 7f6829a33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:01.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.508+0000 7f6829a33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:01.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.508+0000 7f6829a33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:01.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.508+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68241d78a0 con 0x7f68241430c0 2024-03-25T18:24:01.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.543+0000 7f680bfff640 1 -- 172.21.15.162:0/1876797534 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3606349714) 0x7f681c052450 con 0x7f68241430c0 2024-03-25T18:24:01.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.543+0000 7f680bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:01.545 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.543+0000 7f680bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:01.545 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:24:01.545 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":10,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:14:10.037366+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:10.037366+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 10 librados: watch_flush enter 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 10 librados: watch_flush exit 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 10 client.4689.objecter successfully canceled tick 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 >> v1:172.21.15.191:6800/1552703795 conn(0x7f681006dc80 legacy=0x7f6810070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 10 monclient: shutdown 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 >> v1:172.21.15.162:6789/0 conn(0x7f68241430c0 legacy=0x7f68241d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 shutdown_connections 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 >> 172.21.15.162:0/1876797534 conn(0x7f68240b7ea0 msgr2=0x7f6824074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 shutdown_connections 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 1 -- 172.21.15.162:0/1876797534 wait complete. 2024-03-25T18:24:01.548 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:01.546+0000 7f6829a33640 1 librados: shutdown 2024-03-25T18:24:01.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:08.366 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.478+0000 7f77b8c84640 10 monclient: get_monmap_and_config 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.478+0000 7f77b8c84640 10 monclient: build_initial_monmap 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.478+0000 7f77b8c84640 10 monclient: monmap: 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:24:08.479232+0000 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:24:08.479232+0000 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:08.479 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:08.480 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:24:08.480 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:24:08.480 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:08.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.479+0000 7f77b8c84640 1 Processor -- start 2024-03-25T18:24:08.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.479+0000 7f77b8c84640 1 -- start start 2024-03-25T18:24:08.480 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.479+0000 7f77b8c84640 10 monclient: init 2024-03-25T18:24:08.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.479+0000 7f77b8c84640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:08.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.479+0000 7f77b8c84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:08.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.479+0000 7f77b8c84640 10 monclient(hunting): picked mon.noname-c con 0x7f77b406c380 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:08.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.479+0000 7f77b8c84640 10 monclient(hunting): picked mon.noname-b con 0x7f77b40b12d0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:08.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.480+0000 7f77b8c84640 10 monclient(hunting): picked mon.noname-a con 0x7f77b40aedf0 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:08.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.480+0000 7f77b8c84640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40b16f0 con 0x7f77b40aedf0 2024-03-25T18:24:08.481 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.480+0000 7f77b8c84640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40adfe0 con 0x7f77b406c380 2024-03-25T18:24:08.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.480+0000 7f77b8c84640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b40add00 con 0x7f77b40b12d0 2024-03-25T18:24:08.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.480+0000 7f77b8c84640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:08.482 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.480+0000 7f77b8c84640 10 monclient(hunting): authenticate will time out at 8147.753814s 2024-03-25T18:24:08.487 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.486+0000 7f77b259c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f77b406c380 0x7f77b40b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:47270/0 (socket says 172.21.15.162:47270) 2024-03-25T18:24:08.487 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.486+0000 7f77b259c640 1 -- 172.21.15.162:0/3420704254 learned_addr learned my addr 172.21.15.162:0/3420704254 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:08.495 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.494+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292851335 0 0) 0x7f77b40adfe0 con 0x7f77b406c380 2024-03-25T18:24:08.495 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.494+0000 7f77b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:08.495 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.494+0000 7f77b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:08.495 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.494+0000 7f77b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:08.495 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.494+0000 7f77b1d9b640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:24:08.495 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.494+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a0003520 con 0x7f77b406c380 2024-03-25T18:24:08.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2194537069 0 0) 0x7f77a0003520 con 0x7f77b406c380 2024-03-25T18:24:08.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 >> v1:172.21.15.162:6790/0 conn(0x7f77b40b12d0 legacy=0x7f77b40ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 >> v1:172.21.15.162:6789/0 conn(0x7f77b40aedf0 legacy=0x7f77b414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b40ae2c0 con 0x7f77b406c380 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f77a4001690 con 0x7f77b406c380 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b8c84640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b8c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b1d9b640 10 monclient: dump: 2024-03-25T18:24:08.506 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:08.507 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.504+0000 7f77b8c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:08.514 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.513+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77a40017f0 con 0x7f77b406c380 2024-03-25T18:24:08.514 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.513+0000 7f77b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:08.514 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.513+0000 7f77b1d9b640 1 -- 172.21.15.162:0/3420704254 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f77a4001b80 con 0x7f77b406c380 2024-03-25T18:24:08.514 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.513+0000 7f77b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.514 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.513+0000 7f77b1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:08.514 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.513+0000 7f77b1d9b640 10 monclient: dump: 2024-03-25T18:24:08.514 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.513+0000 7f77b8c84640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:08.515 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.513+0000 7f77b8c84640 4 set_mon_vals no callback set 2024-03-25T18:24:08.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.514+0000 7f77b8c84640 10 monclient: shutdown 2024-03-25T18:24:08.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.514+0000 7f77b8c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:08.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.514+0000 7f77b8c84640 1 -- 172.21.15.162:0/3420704254 >> v1:172.21.15.191:6789/0 conn(0x7f77b406c380 legacy=0x7f77b40b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.514+0000 7f77b8c84640 1 -- 172.21.15.162:0/3420704254 shutdown_connections 2024-03-25T18:24:08.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.514+0000 7f77b8c84640 1 -- 172.21.15.162:0/3420704254 >> 172.21.15.162:0/3420704254 conn(0x7f77b40b5d10 msgr2=0x7f77b40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:08.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.514+0000 7f77b8c84640 1 -- 172.21.15.162:0/3420704254 shutdown_connections 2024-03-25T18:24:08.516 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.514+0000 7f77b8c84640 1 -- 172.21.15.162:0/3420704254 wait complete. 2024-03-25T18:24:08.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.515+0000 7f77b8c84640 10 monclient: build_initial_monmap 2024-03-25T18:24:08.517 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.515+0000 7f77b8c84640 10 monclient: monmap: 2024-03-25T18:24:08.517 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:08.517 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:08.517 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:24:08.517 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:24:08.517 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:08.517 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 librados: starting msgr at 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 librados: starting objecter 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 Processor -- start 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 -- start start 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 librados: setting wanted keys 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 librados: calling monclient init 2024-03-25T18:24:08.518 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 10 monclient: init 2024-03-25T18:24:08.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:08.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:08.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 10 monclient(hunting): picked mon.noname-b con 0x7f77b406c380 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:08.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 10 monclient(hunting): picked mon.noname-c con 0x7f77b40aedf0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:08.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 10 monclient(hunting): picked mon.noname-a con 0x7f77b40b12d0 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:08.519 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b406c8e0 con 0x7f77b40b12d0 2024-03-25T18:24:08.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b41c4a20 con 0x7f77b406c380 2024-03-25T18:24:08.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b41c2db0 con 0x7f77b40aedf0 2024-03-25T18:24:08.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:08.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.516+0000 7f77b8c84640 10 monclient(hunting): authenticate will time out at 8147.790600s 2024-03-25T18:24:08.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.519+0000 7f77abfff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f77b40aedf0 0x7f77b41c2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:39026/0 (socket says 172.21.15.162:39026) 2024-03-25T18:24:08.520 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.519+0000 7f77abfff640 1 -- 172.21.15.162:0/4198694508 learned_addr learned my addr 172.21.15.162:0/4198694508 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:08.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.520+0000 7f77aaffd640 10 client.?.objecter ms_handle_connect 0x7f77b40aedf0 2024-03-25T18:24:08.521 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.520+0000 7f77aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:08.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.527+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764632264 0 0) 0x7f77b41c2db0 con 0x7f77b40aedf0 2024-03-25T18:24:08.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.527+0000 7f77aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:08.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.527+0000 7f77aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:08.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.527+0000 7f77aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:08.528 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.527+0000 7f77aaffd640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:24:08.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.527+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7794004eb0 con 0x7f77b40aedf0 2024-03-25T18:24:08.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.533+0000 7f77aaffd640 10 client.?.objecter ms_handle_connect 0x7f77b406c380 2024-03-25T18:24:08.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.533+0000 7f77aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:08.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.534+0000 7f77aaffd640 10 client.?.objecter ms_handle_connect 0x7f77b40b12d0 2024-03-25T18:24:08.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.534+0000 7f77aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.536+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3706253483 0 0) 0x7f7794004eb0 con 0x7f77b40aedf0 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.536+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b41c2db0 con 0x7f77b40aedf0 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.536+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f779c001990 con 0x7f77b40aedf0 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.536+0000 7f77aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.536+0000 7f77aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.536+0000 7f77aaffd640 10 monclient(hunting): dump: 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:08.537 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:08.538 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.538 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:08.538 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:08.538 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:08.538 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:08.538 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:08.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.536+0000 7f77aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:08.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.541+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613022661 0 0) 0x7f77b41c4a20 con 0x7f77b406c380 2024-03-25T18:24:08.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.541+0000 7f77aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:08.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.541+0000 7f77aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:08.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.541+0000 7f77aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:08.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.541+0000 7f77aaffd640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:24:08.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.541+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7794004eb0 con 0x7f77b406c380 2024-03-25T18:24:08.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.546+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921121651 0 0) 0x7f77b406c8e0 con 0x7f77b40b12d0 2024-03-25T18:24:08.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.546+0000 7f77aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:08.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.546+0000 7f77aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:08.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.546+0000 7f77aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:08.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.546+0000 7f77aaffd640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:24:08.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.546+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b41c4a20 con 0x7f77b40b12d0 2024-03-25T18:24:08.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 432077045 0 0) 0x7f77b41c2db0 con 0x7f77b40aedf0 2024-03-25T18:24:08.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 >> v1:172.21.15.191:6789/0 conn(0x7f77b406c380 legacy=0x7f77b41dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 >> v1:172.21.15.162:6789/0 conn(0x7f77b40b12d0 legacy=0x7f77b41c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:08.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 1 monclient: found mon.c 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b41db9d0 con 0x7f77b40aedf0 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 5 monclient: authenticate success, global_id 4673 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 10 monclient: _renew_subs 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b41db510 con 0x7f77b40aedf0 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 10 monclient: _renew_subs 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:24:08.555 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b41c1940 con 0x7f77b40aedf0 2024-03-25T18:24:08.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.553+0000 7f77b8c84640 1 librados: init done 2024-03-25T18:24:08.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.554+0000 7f77b8c84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:08.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.554+0000 7f77b8c84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:08.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.554+0000 7f77b8c84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:24:08.556 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.554+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b406d790 con 0x7f77b40aedf0 2024-03-25T18:24:08.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.845+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f779c003610 con 0x7f77b40aedf0 2024-03-25T18:24:08.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.845+0000 7f77aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:08.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.845+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f779c003800 con 0x7f77b40aedf0 2024-03-25T18:24:08.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.845+0000 7f77aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:08.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.845+0000 7f77aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:08.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.845+0000 7f77aaffd640 10 monclient: dump: 2024-03-25T18:24:08.846 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:08.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.845+0000 7f77b159a640 4 set_mon_vals no callback set 2024-03-25T18:24:08.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.845+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f779c003a20 con 0x7f77b40aedf0 2024-03-25T18:24:08.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.846+0000 7f77b259c640 10 monclient: get_auth_request con 0x7f779406de60 auth_method 0 2024-03-25T18:24:08.848 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.846+0000 7f77aaffd640 10 client.4673.objecter ms_handle_connect 0x7f779406de60 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.851+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1113761810 0 0) 0x7f779c054630 con 0x7f77b40aedf0 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.851+0000 7f77aaffd640 10 client.4673.objecter ms_dispatch 0x7f77b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.851+0000 7f77aaffd640 3 client.4673.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.851+0000 7f77aaffd640 3 client.4673.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.851+0000 7f77aaffd640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.851+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f779c050140 con 0x7f77b40aedf0 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.851+0000 7f77aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:08.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:08.851+0000 7f77aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:09.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.079+0000 7f77b8c84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:09.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.079+0000 7f77b8c84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:09.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.079+0000 7f77b8c84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:24:09.080 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.079+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77b40627b0 con 0x7f77b40aedf0 2024-03-25T18:24:09.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.241+0000 7f77aaffd640 1 -- 172.21.15.162:0/4198694508 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7f779c052070 con 0x7f77b40aedf0 2024-03-25T18:24:09.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.241+0000 7f77aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:09.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.241+0000 7f77aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:09.242 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:24:09.242 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:09.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 10 librados: watch_flush enter 2024-03-25T18:24:09.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 10 librados: watch_flush exit 2024-03-25T18:24:09.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-25T18:24:09.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 10 client.4673.objecter successfully canceled tick 2024-03-25T18:24:09.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 >> v1:172.21.15.191:6800/1552703795 conn(0x7f779406de60 legacy=0x7f7794070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:09.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 10 monclient: shutdown 2024-03-25T18:24:09.245 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:09.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 >> v1:172.21.15.162:6790/0 conn(0x7f77b40aedf0 legacy=0x7f77b41c2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:09.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.243+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 shutdown_connections 2024-03-25T18:24:09.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.244+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 >> 172.21.15.162:0/4198694508 conn(0x7f77b40b5d10 msgr2=0x7f77b40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:09.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.244+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 shutdown_connections 2024-03-25T18:24:09.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.244+0000 7f77b8c84640 1 -- 172.21.15.162:0/4198694508 wait complete. 2024-03-25T18:24:09.246 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:09.244+0000 7f77b8c84640 1 librados: shutdown 2024-03-25T18:24:09.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:16.065 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:16.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.180+0000 7fc47993d640 10 monclient: get_monmap_and_config 2024-03-25T18:24:16.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.180+0000 7fc47993d640 10 monclient: build_initial_monmap 2024-03-25T18:24:16.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.180+0000 7fc47993d640 10 monclient: monmap: 2024-03-25T18:24:16.181 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:16.181 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:16.181 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:24:16.181178+0000 2024-03-25T18:24:16.181 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:24:16.181178+0000 2024-03-25T18:24:16.182 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:16.182 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:16.182 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:16.182 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:24:16.182 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:24:16.182 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:16.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.180+0000 7fc47993d640 1 Processor -- start 2024-03-25T18:24:16.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 1 -- start start 2024-03-25T18:24:16.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 10 monclient: init 2024-03-25T18:24:16.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:16.183 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:16.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 10 monclient(hunting): picked mon.noname-c con 0x7fc47406ab80 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:16.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 10 monclient(hunting): picked mon.noname-b con 0x7fc47406af60 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:16.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 10 monclient(hunting): picked mon.noname-a con 0x7fc4740b00b0 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:16.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740b0500 con 0x7fc4740b00b0 2024-03-25T18:24:16.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc47413ffa0 con 0x7fc47406ab80 2024-03-25T18:24:16.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4740af520 con 0x7fc47406af60 2024-03-25T18:24:16.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:16.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.181+0000 7fc47993d640 10 monclient(hunting): authenticate will time out at 8155.455606s 2024-03-25T18:24:16.188 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.187+0000 7fc472ffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fc47406ab80 0x7fc4740ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:46714/0 (socket says 172.21.15.162:46714) 2024-03-25T18:24:16.188 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.187+0000 7fc472ffd640 1 -- 172.21.15.162:0/3949197713 learned_addr learned my addr 172.21.15.162:0/3949197713 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.200+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490344507 0 0) 0x7fc4740af520 con 0x7fc47406af60 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.200+0000 7fc471ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.200+0000 7fc471ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.200+0000 7fc471ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.200+0000 7fc471ffb640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.200+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc45c003520 con 0x7fc47406af60 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.201+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651949270 0 0) 0x7fc47413ffa0 con 0x7fc47406ab80 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.201+0000 7fc471ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.201+0000 7fc471ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:16.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.201+0000 7fc471ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:16.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.201+0000 7fc471ffb640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:24:16.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.201+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4740af520 con 0x7fc47406ab80 2024-03-25T18:24:16.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.213+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 830015027 0 0) 0x7fc45c003520 con 0x7fc47406af60 2024-03-25T18:24:16.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.213+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 >> v1:172.21.15.191:6789/0 conn(0x7fc47406ab80 legacy=0x7fc4740ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.213+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 >> v1:172.21.15.162:6789/0 conn(0x7fc4740b00b0 legacy=0x7fc47414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.213+0000 7fc471ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:16.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.213+0000 7fc471ffb640 1 monclient: found mon.noname-b 2024-03-25T18:24:16.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:16.215 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.162:6790/0 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc474072890 con 0x7fc47406af60 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc468001690 con 0x7fc47406af60 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc47993d640 5 monclient: authenticate success, global_id 4676 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc47993d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc471ffb640 10 monclient: dump: 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.216 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:16.217 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:16.217 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:16.217 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:16.217 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:16.217 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:16.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.214+0000 7fc47993d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:16.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 <== mon.2 v1:172.21.15.162:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4680017f0 con 0x7fc47406af60 2024-03-25T18:24:16.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc471ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:16.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc471ffb640 1 -- 172.21.15.162:0/3949197713 <== mon.2 v1:172.21.15.162:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc468001c00 con 0x7fc47406af60 2024-03-25T18:24:16.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc47993d640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:16.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc47993d640 4 set_mon_vals no callback set 2024-03-25T18:24:16.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc471ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:16.220 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc471ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc471ffb640 10 monclient: dump: 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:16.221 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:16.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc47993d640 10 monclient: shutdown 2024-03-25T18:24:16.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc47993d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:16.223 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.219+0000 7fc47993d640 1 -- 172.21.15.162:0/3949197713 >> v1:172.21.15.162:6790/0 conn(0x7fc47406af60 legacy=0x7fc47413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.220+0000 7fc47993d640 1 -- 172.21.15.162:0/3949197713 shutdown_connections 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.220+0000 7fc47993d640 1 -- 172.21.15.162:0/3949197713 >> 172.21.15.162:0/3949197713 conn(0x7fc4740b4a80 msgr2=0x7fc4740b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.220+0000 7fc47993d640 1 -- 172.21.15.162:0/3949197713 shutdown_connections 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.220+0000 7fc47993d640 1 -- 172.21.15.162:0/3949197713 wait complete. 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 10 monclient: build_initial_monmap 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 10 monclient: monmap: 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:24:16.224 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 1 librados: starting msgr at 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 1 librados: starting objecter 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 1 Processor -- start 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 1 -- start start 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 1 librados: setting wanted keys 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 1 librados: calling monclient init 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.221+0000 7fc47993d640 10 monclient: init 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:16.225 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 10 monclient(hunting): picked mon.noname-a con 0x7fc47406ab80 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:16.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 10 monclient(hunting): picked mon.noname-c con 0x7fc4740b00b0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:16.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 10 monclient(hunting): picked mon.noname-b con 0x7fc4741448c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:16.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc474144d50 con 0x7fc47406ab80 2024-03-25T18:24:16.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc474138a10 con 0x7fc4741448c0 2024-03-25T18:24:16.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc474137e00 con 0x7fc4740b00b0 2024-03-25T18:24:16.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:16.226 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.222+0000 7fc47993d640 10 monclient(hunting): authenticate will time out at 8155.496193s 2024-03-25T18:24:16.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.229+0000 7fc472ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fc47406ab80 0x7fc474140c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:34010/0 (socket says 172.21.15.162:34010) 2024-03-25T18:24:16.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.229+0000 7fc472ffd640 1 -- 172.21.15.162:0/164148064 learned_addr learned my addr 172.21.15.162:0/164148064 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:16.230 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.229+0000 7fc45bfff640 10 client.?.objecter ms_handle_connect 0x7fc47406ab80 2024-03-25T18:24:16.231 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.229+0000 7fc45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:16.238 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.237+0000 7fc45bfff640 10 client.?.objecter ms_handle_connect 0x7fc4741448c0 2024-03-25T18:24:16.238 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.237+0000 7fc45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:16.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.241+0000 7fc45bfff640 10 client.?.objecter ms_handle_connect 0x7fc4740b00b0 2024-03-25T18:24:16.242 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.241+0000 7fc45bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:16.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.246+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240514061 0 0) 0x7fc474144d50 con 0x7fc47406ab80 2024-03-25T18:24:16.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.246+0000 7fc45bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:16.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.246+0000 7fc45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:16.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.246+0000 7fc45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:16.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.246+0000 7fc45bfff640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:24:16.248 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.246+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc460004eb0 con 0x7fc47406ab80 2024-03-25T18:24:16.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.251+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256985594 0 0) 0x7fc474137e00 con 0x7fc4740b00b0 2024-03-25T18:24:16.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.251+0000 7fc45bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:16.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.251+0000 7fc45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:16.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.251+0000 7fc45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:16.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.251+0000 7fc45bfff640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:24:16.252 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.251+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc474144d50 con 0x7fc4740b00b0 2024-03-25T18:24:16.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.256+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357562152 0 0) 0x7fc474138a10 con 0x7fc4741448c0 2024-03-25T18:24:16.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.256+0000 7fc45bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:16.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.256+0000 7fc45bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:16.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.256+0000 7fc45bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:16.257 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.256+0000 7fc45bfff640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:24:16.258 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.256+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc474137e00 con 0x7fc4741448c0 2024-03-25T18:24:16.271 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.270+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3653836079 0 0) 0x7fc460004eb0 con 0x7fc47406ab80 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.270+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc474138a10 con 0x7fc47406ab80 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.270+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc4640019f0 con 0x7fc47406ab80 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.270+0000 7fc45bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.270+0000 7fc45bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.270+0000 7fc45bfff640 10 monclient(hunting): dump: 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:16.272 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:16.273 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.270+0000 7fc45bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.274+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 55364484 0 0) 0x7fc474137e00 con 0x7fc4741448c0 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.274+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc460004eb0 con 0x7fc4741448c0 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.274+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc46c001ad0 con 0x7fc4741448c0 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.274+0000 7fc45bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.274+0000 7fc45bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.274+0000 7fc45bfff640 10 monclient(hunting): dump: 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.275 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.276 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:16.276 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:16.276 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:16.276 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:16.276 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:16.276 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:16.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.280+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47481218 0 0) 0x7fc474144d50 con 0x7fc4740b00b0 2024-03-25T18:24:16.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.280+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc474137e00 con 0x7fc4740b00b0 2024-03-25T18:24:16.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.280+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc468001b00 con 0x7fc4740b00b0 2024-03-25T18:24:16.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.280+0000 7fc45bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:16.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.280+0000 7fc45bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:16.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.280+0000 7fc45bfff640 10 monclient(hunting): dump: 2024-03-25T18:24:16.281 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:16.282 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:16.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.283+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2770682280 0 0) 0x7fc460004eb0 con 0x7fc4741448c0 2024-03-25T18:24:16.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.283+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 >> v1:172.21.15.162:6790/0 conn(0x7fc4740b00b0 legacy=0x7fc4741376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 >> v1:172.21.15.162:6789/0 conn(0x7fc47406ab80 legacy=0x7fc474140c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 1 monclient: found mon.b 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc474137f90 con 0x7fc4741448c0 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 10 monclient: _finish_auth 0 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc45bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 10 monclient: _renew_subs 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4741db480 con 0x7fc4741448c0 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:16.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 10 monclient: _renew_subs 2024-03-25T18:24:16.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:16.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc474146410 con 0x7fc4741448c0 2024-03-25T18:24:16.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.284+0000 7fc47993d640 1 librados: init done 2024-03-25T18:24:16.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.285+0000 7fc47993d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:16.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.285+0000 7fc47993d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:16.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.285+0000 7fc47993d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:16.287 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.285+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4740680d0 con 0x7fc4741448c0 2024-03-25T18:24:16.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.559+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc46c001fe0 con 0x7fc4741448c0 2024-03-25T18:24:16.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.559+0000 7fc45bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:16.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.559+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fc46c003b10 con 0x7fc4741448c0 2024-03-25T18:24:16.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.559+0000 7fc45bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:16.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.559+0000 7fc45bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:16.561 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.559+0000 7fc45bfff640 10 monclient: dump: 2024-03-25T18:24:16.561 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:16.561 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:16.562 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.559+0000 7fc4717fa640 4 set_mon_vals no callback set 2024-03-25T18:24:16.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.560+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fc46c019ae0 con 0x7fc4741448c0 2024-03-25T18:24:16.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.561+0000 7fc472ffd640 10 monclient: get_auth_request con 0x7fc46006e0c0 auth_method 0 2024-03-25T18:24:16.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.561+0000 7fc45bfff640 10 client.4675.objecter ms_handle_connect 0x7fc46006e0c0 2024-03-25T18:24:16.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.568+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7fc46c053830 con 0x7fc4741448c0 2024-03-25T18:24:16.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.568+0000 7fc45bfff640 10 client.4675.objecter ms_dispatch 0x7fc474000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:16.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.568+0000 7fc45bfff640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:16.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.568+0000 7fc45bfff640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:16.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.568+0000 7fc45bfff640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:24:16.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.568+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc46c04f170 con 0x7fc4741448c0 2024-03-25T18:24:16.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.568+0000 7fc45bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:16.570 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.568+0000 7fc45bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:16.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.799+0000 7fc47993d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:16.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.799+0000 7fc47993d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:16.800 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.799+0000 7fc47993d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:16.801 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.799+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc474138120 con 0x7fc4741448c0 2024-03-25T18:24:16.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.959+0000 7fc45bfff640 1 -- 172.21.15.162:0/164148064 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7fc46c051270 con 0x7fc4741448c0 2024-03-25T18:24:16.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.959+0000 7fc45bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:16.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.959+0000 7fc45bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:16.961 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:24:16.961 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:16.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 10 librados: watch_flush enter 2024-03-25T18:24:16.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 10 librados: watch_flush exit 2024-03-25T18:24:16.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:24:16.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:24:16.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 >> v1:172.21.15.191:6800/1552703795 conn(0x7fc46006e0c0 legacy=0x7fc460070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 10 monclient: shutdown 2024-03-25T18:24:16.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:16.963 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 >> v1:172.21.15.191:6789/0 conn(0x7fc4741448c0 legacy=0x7fc4741e35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:16.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 shutdown_connections 2024-03-25T18:24:16.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.961+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 >> 172.21.15.162:0/164148064 conn(0x7fc4740b4a80 msgr2=0x7fc4740ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:16.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.962+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 shutdown_connections 2024-03-25T18:24:16.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.962+0000 7fc47993d640 1 -- 172.21.15.162:0/164148064 wait complete. 2024-03-25T18:24:16.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:16.962+0000 7fc47993d640 1 librados: shutdown 2024-03-25T18:24:16.973 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:23.780 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:23.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.893+0000 7fb2c9123640 10 monclient: get_monmap_and_config 2024-03-25T18:24:23.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.894+0000 7fb2c9123640 10 monclient: build_initial_monmap 2024-03-25T18:24:23.895 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.894+0000 7fb2c9123640 10 monclient: monmap: 2024-03-25T18:24:23.895 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:23.895 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:23.895 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:24:23.895061+0000 2024-03-25T18:24:23.895 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:24:23.895061+0000 2024-03-25T18:24:23.896 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:23.896 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:23.896 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:23.896 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:24:23.896 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:24:23.896 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:23.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.894+0000 7fb2c9123640 1 Processor -- start 2024-03-25T18:24:23.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.894+0000 7fb2c9123640 1 -- start start 2024-03-25T18:24:23.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.894+0000 7fb2c9123640 10 monclient: init 2024-03-25T18:24:23.897 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c40a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c41430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c4142520 con 0x7fb2c413f710 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40b28a0 con 0x7fb2c41430c0 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40b2a30 con 0x7fb2c40a98f0 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:23.898 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.895+0000 7fb2c9123640 10 monclient(hunting): authenticate will time out at 8163.169301s 2024-03-25T18:24:23.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.899+0000 7fb2c359e640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fb2c41430c0 0x7fb2c414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:35774/0 (socket says 172.21.15.162:35774) 2024-03-25T18:24:23.900 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.899+0000 7fb2c359e640 1 -- 172.21.15.162:0/928564923 learned_addr learned my addr 172.21.15.162:0/928564923 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:23.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.910+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521390140 0 0) 0x7fb2c40b28a0 con 0x7fb2c41430c0 2024-03-25T18:24:23.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.911+0000 7fb2c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:23.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.911+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:23.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.911+0000 7fb2c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:23.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.911+0000 7fb2c1d9b640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:24:23.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.911+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ac003520 con 0x7fb2c41430c0 2024-03-25T18:24:23.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3192732257 0 0) 0x7fb2ac003520 con 0x7fb2c41430c0 2024-03-25T18:24:23.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 >> v1:172.21.15.162:6790/0 conn(0x7fb2c40a98f0 legacy=0x7fb2c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 >> v1:172.21.15.162:6789/0 conn(0x7fb2c413f710 legacy=0x7fb2c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c40b44c0 con 0x7fb2c41430c0 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2bc001a10 con 0x7fb2c41430c0 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c1d9b640 10 monclient: dump: 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:23.926 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c9123640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:24:23.927 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.923+0000 7fb2c9123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.927+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2bc001be0 con 0x7fb2c41430c0 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.927+0000 7fb2c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.927+0000 7fb2c1d9b640 1 -- 172.21.15.162:0/928564923 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2bc002030 con 0x7fb2c41430c0 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.927+0000 7fb2c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.927+0000 7fb2c1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.927+0000 7fb2c1d9b640 10 monclient: dump: 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:23.929 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.930 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:23.930 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:23.930 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:23.930 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:23.930 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:23.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.927+0000 7fb2c9123640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:23.930 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.927+0000 7fb2c9123640 4 set_mon_vals no callback set 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.928+0000 7fb2c9123640 10 monclient: shutdown 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.928+0000 7fb2c9123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.928+0000 7fb2c9123640 1 -- 172.21.15.162:0/928564923 >> v1:172.21.15.191:6789/0 conn(0x7fb2c41430c0 legacy=0x7fb2c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.928+0000 7fb2c9123640 1 -- 172.21.15.162:0/928564923 shutdown_connections 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.928+0000 7fb2c9123640 1 -- 172.21.15.162:0/928564923 >> 172.21.15.162:0/928564923 conn(0x7fb2c40b7ea0 msgr2=0x7fb2c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.928+0000 7fb2c9123640 1 -- 172.21.15.162:0/928564923 shutdown_connections 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.928+0000 7fb2c9123640 1 -- 172.21.15.162:0/928564923 wait complete. 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.929+0000 7fb2c9123640 10 monclient: build_initial_monmap 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.929+0000 7fb2c9123640 10 monclient: monmap: 2024-03-25T18:24:23.933 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.930+0000 7fb2c9123640 1 librados: starting msgr at 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.930+0000 7fb2c9123640 1 librados: starting objecter 2024-03-25T18:24:23.934 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.930+0000 7fb2c9123640 1 Processor -- start 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.930+0000 7fb2c9123640 1 -- start start 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.930+0000 7fb2c9123640 1 librados: setting wanted keys 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.930+0000 7fb2c9123640 1 librados: calling monclient init 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.930+0000 7fb2c9123640 10 monclient: init 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.931+0000 7fb2c9123640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.931+0000 7fb2c9123640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.931+0000 7fb2c9123640 10 monclient(hunting): picked mon.noname-c con 0x7fb2c40a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.931+0000 7fb2c9123640 10 monclient(hunting): picked mon.noname-a con 0x7fb2c413f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.931+0000 7fb2c9123640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c41430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.931+0000 7fb2c9123640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c4135510 con 0x7fb2c413f710 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.931+0000 7fb2c9123640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c40aba50 con 0x7fb2c41430c0 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.932+0000 7fb2c9123640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2c41351a0 con 0x7fb2c40a98f0 2024-03-25T18:24:23.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.932+0000 7fb2c9123640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:23.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.932+0000 7fb2c9123640 10 monclient(hunting): authenticate will time out at 8163.205798s 2024-03-25T18:24:23.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.935+0000 7fb2c2d9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7fb2c40a98f0 0x7fb2c41359a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:54380/0 (socket says 172.21.15.162:54380) 2024-03-25T18:24:23.936 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.935+0000 7fb2c2d9d640 1 -- 172.21.15.162:0/2671355218 learned_addr learned my addr 172.21.15.162:0/2671355218 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:23.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.938+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c40a98f0 2024-03-25T18:24:23.939 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.938+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:23.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.945+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c413f710 2024-03-25T18:24:23.946 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.945+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:23.949 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.948+0000 7fb2ab7fe640 10 client.?.objecter ms_handle_connect 0x7fb2c41430c0 2024-03-25T18:24:23.949 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.948+0000 7fb2ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:23.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.951+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111197198 0 0) 0x7fb2c4135510 con 0x7fb2c413f710 2024-03-25T18:24:23.952 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.951+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.951+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.951+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.951+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:24:23.953 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:23.951+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b0004eb0 con 0x7fb2c413f710 2024-03-25T18:24:24.009 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:36.237 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.014 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.013+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2282568566 0 0) 0x7fb2b0004eb0 con 0x7fb2c413f710 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.013+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c4135510 con 0x7fb2c413f710 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.013+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2b8001870 con 0x7fb2c413f710 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.013+0000 7fb2ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.013+0000 7fb2ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.013+0000 7fb2ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:24.015 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:24.016 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:24.016 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:24.016 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.013+0000 7fb2ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:24.029 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:36.257 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.063 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:36.291 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.072+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809332816 0 0) 0x7fb2c40aba50 con 0x7fb2c41430c0 2024-03-25T18:24:24.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.072+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:24.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.072+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:24.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.072+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:24.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.072+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:24:24.074 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.072+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b0004eb0 con 0x7fb2c41430c0 2024-03-25T18:24:24.075 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:36.291 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.083 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:36.299 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.087 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:36.315 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.088+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2317189134 0 0) 0x7fb2b0004eb0 con 0x7fb2c41430c0 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.088+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2c40aba50 con 0x7fb2c41430c0 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.088+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2bc001d50 con 0x7fb2c41430c0 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.088+0000 7fb2ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.088+0000 7fb2ab7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.088+0000 7fb2ab7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:24.090 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.089+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435259849 0 0) 0x7fb2c41351a0 con 0x7fb2c40a98f0 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.089+0000 7fb2ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.089+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.089+0000 7fb2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:24.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.089+0000 7fb2ab7fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:24:24.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.089+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2b0004eb0 con 0x7fb2c40a98f0 2024-03-25T18:24:24.092 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:36.320 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3036305239 0 0) 0x7fb2c4135510 con 0x7fb2c413f710 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 >> v1:172.21.15.162:6790/0 conn(0x7fb2c40a98f0 legacy=0x7fb2c41359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 >> v1:172.21.15.191:6789/0 conn(0x7fb2c41430c0 legacy=0x7fb2c41e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 1 monclient: found mon.a 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c4137c80 con 0x7fb2c413f710 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:24.103 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 5 monclient: authenticate success, global_id 4704 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 10 monclient: _renew_subs 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c4137ed0 con 0x7fb2c413f710 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 10 client.4704.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 10 monclient: _renew_subs 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c4138690 con 0x7fb2c413f710 2024-03-25T18:24:24.104 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.101+0000 7fb2c9123640 1 librados: init done 2024-03-25T18:24:24.105 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.103+0000 7fb2c9123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:24.105 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.103+0000 7fb2c9123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:24.105 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.103+0000 7fb2c9123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:24.105 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.103+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c40b3ef0 con 0x7fb2c413f710 2024-03-25T18:24:24.109 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:36.325 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.133 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:36.361 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.499+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2b80035f0 con 0x7fb2c413f710 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.499+0000 7fb2ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.499+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fb2b8003900 con 0x7fb2c413f710 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.499+0000 7fb2ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.499+0000 7fb2ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.499+0000 7fb2ab7fe640 10 monclient: dump: 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:24.501 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:24.502 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:24.502 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:24.502 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:24.502 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:24.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.499+0000 7fb2c159a640 4 set_mon_vals no callback set 2024-03-25T18:24:24.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.500+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fb2b80198d0 con 0x7fb2c413f710 2024-03-25T18:24:24.502 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.501+0000 7fb2c2d9d640 10 monclient: get_auth_request con 0x7fb2b006df20 auth_method 0 2024-03-25T18:24:24.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.501+0000 7fb2ab7fe640 10 client.4704.objecter ms_handle_connect 0x7fb2b006df20 2024-03-25T18:24:24.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.508+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7fb2b80535a0 con 0x7fb2c413f710 2024-03-25T18:24:24.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.508+0000 7fb2ab7fe640 10 client.4704.objecter ms_dispatch 0x7fb2c4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:24.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.508+0000 7fb2ab7fe640 3 client.4704.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:24.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.508+0000 7fb2ab7fe640 3 client.4704.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:24.509 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.508+0000 7fb2ab7fe640 20 client.4704.objecter dump_active .. 0 homeless 2024-03-25T18:24:24.510 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.508+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2b804eee0 con 0x7fb2c413f710 2024-03-25T18:24:24.510 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.508+0000 7fb2ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:24.510 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.508+0000 7fb2ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:24.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.758+0000 7fb2c9123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:24.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.758+0000 7fb2c9123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:24.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.758+0000 7fb2c9123640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:24:24.759 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.758+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2c41356c0 con 0x7fb2c413f710 2024-03-25T18:24:24.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.797+0000 7fb2ab7fe640 1 -- 172.21.15.162:0/2671355218 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7fb2c41356c0 con 0x7fb2c413f710 2024-03-25T18:24:24.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.797+0000 7fb2ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:24.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.797+0000 7fb2ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:24.799 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:24:24.799 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:24.802 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 10 librados: watch_flush enter 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 10 librados: watch_flush exit 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 20 client.4704.objecter shutdown clearing up homeless session... 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 10 client.4704.objecter successfully canceled tick 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 >> v1:172.21.15.191:6800/1552703795 conn(0x7fb2b006df20 legacy=0x7fb2b00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 10 monclient: shutdown 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 >> v1:172.21.15.162:6789/0 conn(0x7fb2c413f710 legacy=0x7fb2c41360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 shutdown_connections 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.799+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 >> 172.21.15.162:0/2671355218 conn(0x7fb2c40b7ea0 msgr2=0x7fb2c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.800+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 shutdown_connections 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.800+0000 7fb2c9123640 1 -- 172.21.15.162:0/2671355218 wait complete. 2024-03-25T18:24:24.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:24.800+0000 7fb2c9123640 1 librados: shutdown 2024-03-25T18:24:24.813 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:31.618 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.731+0000 7f571f663640 10 monclient: get_monmap_and_config 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.731+0000 7f571f663640 10 monclient: build_initial_monmap 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.731+0000 7f571f663640 10 monclient: monmap: 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:24:31.732897+0000 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:24:31.732897+0000 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:31.733 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:31.734 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:24:31.734 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:24:31.734 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:31.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.733+0000 7f571f663640 1 Processor -- start 2024-03-25T18:24:31.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.733+0000 7f571f663640 1 -- start start 2024-03-25T18:24:31.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.733+0000 7f571f663640 10 monclient: init 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 10 monclient(hunting): picked mon.noname-b con 0x7f57180a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 10 monclient(hunting): picked mon.noname-c con 0x7f571813f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 10 monclient(hunting): picked mon.noname-a con 0x7f57181430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5718142520 con 0x7f57181430c0 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57180b28a0 con 0x7f571813f710 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57180b2a30 con 0x7f57180a98f0 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:31.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.734+0000 7f571f663640 10 monclient(hunting): authenticate will time out at 8171.008395s 2024-03-25T18:24:31.750 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.749+0000 7f571cbfe640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f571813f710 0x7f57181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:40476/0 (socket says 172.21.15.162:40476) 2024-03-25T18:24:31.750 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.749+0000 7f571cbfe640 1 -- 172.21.15.162:0/1366272420 learned_addr learned my addr 172.21.15.162:0/1366272420 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:31.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.767+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195694205 0 0) 0x7f57180b28a0 con 0x7f571813f710 2024-03-25T18:24:31.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.767+0000 7f570ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.767+0000 7f570ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.767+0000 7f570ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.767+0000 7f570ffff640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:24:31.768 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.767+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5704003520 con 0x7f571813f710 2024-03-25T18:24:31.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.769+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518914827 0 0) 0x7f57180b2a30 con 0x7f57180a98f0 2024-03-25T18:24:31.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.770+0000 7f570ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.770+0000 7f570ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.770+0000 7f570ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.770+0000 7f570ffff640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:24:31.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.770+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57180b28a0 con 0x7f57180a98f0 2024-03-25T18:24:31.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.775+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756932555 0 0) 0x7f5718142520 con 0x7f57181430c0 2024-03-25T18:24:31.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.775+0000 7f570ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.775+0000 7f570ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.775+0000 7f570ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.775+0000 7f570ffff640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:24:31.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.775+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57180b2a30 con 0x7f57181430c0 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4280876329 0 0) 0x7f5704003520 con 0x7f571813f710 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 >> v1:172.21.15.162:6790/0 conn(0x7f57180a98f0 legacy=0x7f57180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 >> v1:172.21.15.162:6789/0 conn(0x7f57181430c0 legacy=0x7f571814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 1 monclient: found mon.noname-c 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180b44c0 con 0x7f571813f710 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:31.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5714001690 con 0x7f571813f710 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f570ffff640 10 monclient: dump: 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:31.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f571f663640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:24:31.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.779+0000 7f571f663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.790+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57140017f0 con 0x7f571813f710 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.790+0000 7f570ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.790+0000 7f570ffff640 1 -- 172.21.15.162:0/1366272420 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5714001c00 con 0x7f571813f710 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.790+0000 7f570ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.790+0000 7f570ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.790+0000 7f570ffff640 10 monclient: dump: 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.792 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.790+0000 7f571f663640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:31.793 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.790+0000 7f571f663640 4 set_mon_vals no callback set 2024-03-25T18:24:31.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.791+0000 7f571f663640 10 monclient: shutdown 2024-03-25T18:24:31.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.791+0000 7f571f663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:31.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.791+0000 7f571f663640 1 -- 172.21.15.162:0/1366272420 >> v1:172.21.15.191:6789/0 conn(0x7f571813f710 legacy=0x7f57181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.792+0000 7f571f663640 1 -- 172.21.15.162:0/1366272420 shutdown_connections 2024-03-25T18:24:31.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.792+0000 7f571f663640 1 -- 172.21.15.162:0/1366272420 >> 172.21.15.162:0/1366272420 conn(0x7f57180b7ea0 msgr2=0x7f57180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:31.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.792+0000 7f571f663640 1 -- 172.21.15.162:0/1366272420 shutdown_connections 2024-03-25T18:24:31.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.792+0000 7f571f663640 1 -- 172.21.15.162:0/1366272420 wait complete. 2024-03-25T18:24:31.794 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 10 monclient: build_initial_monmap 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 10 monclient: monmap: 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:24:31.795 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:31.796 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 1 librados: starting msgr at 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 1 librados: starting objecter 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 1 Processor -- start 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 1 -- start start 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 1 librados: setting wanted keys 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 1 librados: calling monclient init 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.793+0000 7f571f663640 10 monclient: init 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 10 monclient(hunting): picked mon.noname-b con 0x7f57180a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 10 monclient(hunting): picked mon.noname-c con 0x7f571813f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 10 monclient(hunting): picked mon.noname-a con 0x7f57181430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5718139730 con 0x7f57181430c0 2024-03-25T18:24:31.797 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5718134de0 con 0x7f57180a98f0 2024-03-25T18:24:31.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5718134f70 con 0x7f571813f710 2024-03-25T18:24:31.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:31.798 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.794+0000 7f571f663640 10 monclient(hunting): authenticate will time out at 8171.068633s 2024-03-25T18:24:31.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.802+0000 7f571cbfe640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f571813f710 0x7f57181d7aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:59236/0 (socket says 172.21.15.162:59236) 2024-03-25T18:24:31.803 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.802+0000 7f571cbfe640 1 -- 172.21.15.162:0/1211684453 learned_addr learned my addr 172.21.15.162:0/1211684453 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:31.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.806+0000 7f570dffb640 10 client.?.objecter ms_handle_connect 0x7f571813f710 2024-03-25T18:24:31.807 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.806+0000 7f570dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:31.811 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.810+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138783220 0 0) 0x7f5718134f70 con 0x7f571813f710 2024-03-25T18:24:31.812 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.810+0000 7f570dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.812 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.810+0000 7f570dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.812 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.810+0000 7f570dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.812 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.810+0000 7f570dffb640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:24:31.812 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.811+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56fc004eb0 con 0x7f571813f710 2024-03-25T18:24:31.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.814+0000 7f570dffb640 10 client.?.objecter ms_handle_connect 0x7f57180a98f0 2024-03-25T18:24:31.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.814+0000 7f570dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:31.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.819+0000 7f570dffb640 10 client.?.objecter ms_handle_connect 0x7f57181430c0 2024-03-25T18:24:31.820 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.819+0000 7f570dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:31.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.820+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570118370 0 0) 0x7f5718134de0 con 0x7f57180a98f0 2024-03-25T18:24:31.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.820+0000 7f570dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.820+0000 7f570dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.820+0000 7f570dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.820+0000 7f570dffb640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:24:31.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.820+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5718134f70 con 0x7f57180a98f0 2024-03-25T18:24:31.822 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.821+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1211254330 0 0) 0x7f56fc004eb0 con 0x7f571813f710 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.821+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5718134de0 con 0x7f571813f710 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.821+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5714001bc0 con 0x7f571813f710 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.821+0000 7f570dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.821+0000 7f570dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.821+0000 7f570dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:31.823 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:31.824 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:31.824 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.821+0000 7f570dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:31.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.833+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2625228546 0 0) 0x7f5718134f70 con 0x7f57180a98f0 2024-03-25T18:24:31.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.833+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56fc004eb0 con 0x7f57180a98f0 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.833+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5710001cf0 con 0x7f57180a98f0 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.833+0000 7f570dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.833+0000 7f570dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.833+0000 7f570dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:31.835 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:31.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.837+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201887459 0 0) 0x7f5718139730 con 0x7f57181430c0 2024-03-25T18:24:31.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.837+0000 7f570dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:31.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.837+0000 7f570dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:31.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.837+0000 7f570dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:31.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.837+0000 7f570dffb640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:24:31.838 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.837+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5718134f70 con 0x7f57181430c0 2024-03-25T18:24:31.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1916722162 0 0) 0x7f56fc004eb0 con 0x7f57180a98f0 2024-03-25T18:24:31.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 >> v1:172.21.15.162:6790/0 conn(0x7f571813f710 legacy=0x7f57181d7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 >> v1:172.21.15.162:6789/0 conn(0x7f57181430c0 legacy=0x7f57181e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:31.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 1 monclient: found mon.b 2024-03-25T18:24:31.845 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5718138b20 con 0x7f57180a98f0 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f570dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f571f663640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f571f663640 10 monclient: _renew_subs 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f571f663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.843+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57181d81b0 con 0x7f57180a98f0 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.844+0000 7f571f663640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.844+0000 7f571f663640 10 monclient: _renew_subs 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.844+0000 7f571f663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.844+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5718137870 con 0x7f57180a98f0 2024-03-25T18:24:31.846 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.844+0000 7f571f663640 1 librados: init done 2024-03-25T18:24:31.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.845+0000 7f571f663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:31.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.845+0000 7f571f663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:31.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.845+0000 7f571f663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:31.847 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:31.845+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57180627b0 con 0x7f57180a98f0 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.121+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5710003990 con 0x7f57180a98f0 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.121+0000 7f570dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.121+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f5710003ca0 con 0x7f57180a98f0 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.121+0000 7f570dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.121+0000 7f570dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.121+0000 7f570dffb640 10 monclient: dump: 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:32.123 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:32.124 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:32.124 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:32.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.121+0000 7f570f7fe640 4 set_mon_vals no callback set 2024-03-25T18:24:32.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.122+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f5710019c70 con 0x7f57180a98f0 2024-03-25T18:24:32.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.123+0000 7f571cbfe640 10 monclient: get_auth_request con 0x7f56fc06df30 auth_method 0 2024-03-25T18:24:32.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.123+0000 7f570dffb640 10 client.4687.objecter ms_handle_connect 0x7f56fc06df30 2024-03-25T18:24:32.128 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.126+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7f5710053970 con 0x7f57180a98f0 2024-03-25T18:24:32.128 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.126+0000 7f570dffb640 10 client.4687.objecter ms_dispatch 0x7f5718000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:32.128 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.126+0000 7f570dffb640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:32.128 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.126+0000 7f570dffb640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:32.128 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.126+0000 7f570dffb640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T18:24:32.128 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.127+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f571004f2b0 con 0x7f57180a98f0 2024-03-25T18:24:32.128 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.127+0000 7f570dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:32.128 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.127+0000 7f570dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:32.376 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.375+0000 7f571f663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:32.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.375+0000 7f571f663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:32.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.375+0000 7f571f663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:32.377 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.375+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f571814f300 con 0x7f57180a98f0 2024-03-25T18:24:32.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.404+0000 7f570dffb640 1 -- 172.21.15.162:0/1211684453 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7f57100513b0 con 0x7f57180a98f0 2024-03-25T18:24:32.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.404+0000 7f570dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:32.405 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.404+0000 7f570dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:32.405 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:24:32.405 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:32.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.406+0000 7f571f663640 10 librados: watch_flush enter 2024-03-25T18:24:32.408 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 10 librados: watch_flush exit 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 10 client.4687.objecter successfully canceled tick 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 >> v1:172.21.15.191:6800/1552703795 conn(0x7f56fc06df30 legacy=0x7f56fc0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 10 monclient: shutdown 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 >> v1:172.21.15.191:6789/0 conn(0x7f57180a98f0 legacy=0x7f5718138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 shutdown_connections 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 >> 172.21.15.162:0/1211684453 conn(0x7f57180b7ea0 msgr2=0x7f5718074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 shutdown_connections 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 1 -- 172.21.15.162:0/1211684453 wait complete. 2024-03-25T18:24:32.409 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:32.407+0000 7f571f663640 1 librados: shutdown 2024-03-25T18:24:32.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:39.227 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.337+0000 7fcfa68b5640 10 monclient: get_monmap_and_config 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.337+0000 7fcfa68b5640 10 monclient: build_initial_monmap 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.338+0000 7fcfa68b5640 10 monclient: monmap: 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:24:39.339029+0000 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:24:39.339029+0000 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:39.339 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:24:39.340 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:24:39.340 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.338+0000 7fcfa68b5640 1 Processor -- start 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.338+0000 7fcfa68b5640 1 -- start start 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.338+0000 7fcfa68b5640 10 monclient: init 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa00a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa013f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa01430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa0142520 con 0x7fcfa01430c0 2024-03-25T18:24:39.341 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa00b28a0 con 0x7fcfa013f710 2024-03-25T18:24:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa00b2a30 con 0x7fcfa00a98f0 2024-03-25T18:24:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:39.342 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.339+0000 7fcfa68b5640 10 monclient(hunting): authenticate will time out at 8178.613237s 2024-03-25T18:24:39.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.344+0000 7fcfa4e52640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fcfa01430c0 0x7fcfa014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:59302/0 (socket says 172.21.15.162:59302) 2024-03-25T18:24:39.345 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.344+0000 7fcfa4e52640 1 -- 172.21.15.162:0/578708336 learned_addr learned my addr 172.21.15.162:0/578708336 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:39.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.354+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577560898 0 0) 0x7fcfa0142520 con 0x7fcfa01430c0 2024-03-25T18:24:39.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.354+0000 7fcf9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:39.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.354+0000 7fcf9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:39.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.354+0000 7fcf9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:39.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.354+0000 7fcf9effd640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:24:39.355 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.354+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf88003520 con 0x7fcfa01430c0 2024-03-25T18:24:39.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.361+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191353066 0 0) 0x7fcfa00b28a0 con 0x7fcfa013f710 2024-03-25T18:24:39.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.361+0000 7fcf9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:39.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.361+0000 7fcf9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:39.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.361+0000 7fcf9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:39.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.361+0000 7fcf9effd640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:24:39.362 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.361+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa0142520 con 0x7fcfa013f710 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 819109318 0 0) 0x7fcf88003520 con 0x7fcfa01430c0 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 >> v1:172.21.15.162:6790/0 conn(0x7fcfa00a98f0 legacy=0x7fcfa00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 >> v1:172.21.15.191:6789/0 conn(0x7fcfa013f710 legacy=0x7fcfa01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 1 monclient: found mon.noname-a 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa00b44c0 con 0x7fcfa01430c0 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 10 monclient: _finish_auth 0 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:39.366 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcf98001a10 con 0x7fcfa01430c0 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcf9effd640 10 monclient: dump: 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:39.367 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcfa68b5640 5 monclient: authenticate success, global_id 4713 2024-03-25T18:24:39.368 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.365+0000 7fcfa68b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:39.379 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.378+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf98001b70 con 0x7fcfa01430c0 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.378+0000 7fcf9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.378+0000 7fcf9effd640 1 -- 172.21.15.162:0/578708336 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcf98001f80 con 0x7fcfa01430c0 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.378+0000 7fcf9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.378+0000 7fcf9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.378+0000 7fcf9effd640 10 monclient: dump: 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:39.380 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:39.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.378+0000 7fcfa68b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:39.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.378+0000 7fcfa68b5640 4 set_mon_vals no callback set 2024-03-25T18:24:39.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.379+0000 7fcfa68b5640 10 monclient: shutdown 2024-03-25T18:24:39.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.379+0000 7fcfa68b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:39.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.379+0000 7fcfa68b5640 1 -- 172.21.15.162:0/578708336 >> v1:172.21.15.162:6789/0 conn(0x7fcfa01430c0 legacy=0x7fcfa014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.379+0000 7fcfa68b5640 1 -- 172.21.15.162:0/578708336 shutdown_connections 2024-03-25T18:24:39.381 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.380+0000 7fcfa68b5640 1 -- 172.21.15.162:0/578708336 >> 172.21.15.162:0/578708336 conn(0x7fcfa00b7ea0 msgr2=0x7fcfa00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:39.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.380+0000 7fcfa68b5640 1 -- 172.21.15.162:0/578708336 shutdown_connections 2024-03-25T18:24:39.382 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.380+0000 7fcfa68b5640 1 -- 172.21.15.162:0/578708336 wait complete. 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.381+0000 7fcfa68b5640 10 monclient: build_initial_monmap 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.381+0000 7fcfa68b5640 10 monclient: monmap: 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:39.383 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.381+0000 7fcfa68b5640 1 librados: starting msgr at 2024-03-25T18:24:39.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.381+0000 7fcfa68b5640 1 librados: starting objecter 2024-03-25T18:24:39.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.381+0000 7fcfa68b5640 1 Processor -- start 2024-03-25T18:24:39.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.381+0000 7fcfa68b5640 1 -- start start 2024-03-25T18:24:39.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 1 librados: setting wanted keys 2024-03-25T18:24:39.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 1 librados: calling monclient init 2024-03-25T18:24:39.384 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 10 monclient: init 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa00a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa013f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa01430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa0136c50 con 0x7fcfa00a98f0 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa0136ec0 con 0x7fcfa013f710 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa01317f0 con 0x7fcfa01430c0 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:39.385 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.382+0000 7fcfa68b5640 10 monclient(hunting): authenticate will time out at 8178.656446s 2024-03-25T18:24:39.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.398+0000 7fcf9ffff640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fcfa00a98f0 0x7fcfa01310e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:59330/0 (socket says 172.21.15.162:59330) 2024-03-25T18:24:39.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.398+0000 7fcf9ffff640 1 -- 172.21.15.162:0/3988655305 learned_addr learned my addr 172.21.15.162:0/3988655305 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:39.399 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.399+0000 7fcf9cff9640 10 client.?.objecter ms_handle_connect 0x7fcfa00a98f0 2024-03-25T18:24:39.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.399+0000 7fcf9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:39.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.399+0000 7fcf9cff9640 10 client.?.objecter ms_handle_connect 0x7fcfa013f710 2024-03-25T18:24:39.400 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.399+0000 7fcf9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:39.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.403+0000 7fcf9cff9640 10 client.?.objecter ms_handle_connect 0x7fcfa01430c0 2024-03-25T18:24:39.404 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.403+0000 7fcf9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:39.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.409+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033430368 0 0) 0x7fcfa0136ec0 con 0x7fcfa013f710 2024-03-25T18:24:39.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.409+0000 7fcf9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:39.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.409+0000 7fcf9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:39.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.409+0000 7fcf9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:39.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.409+0000 7fcf9cff9640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:24:39.410 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.409+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf8c004eb0 con 0x7fcfa013f710 2024-03-25T18:24:39.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830303230 0 0) 0x7fcfa0136c50 con 0x7fcfa00a98f0 2024-03-25T18:24:39.411 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa0136ec0 con 0x7fcfa00a98f0 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072788483 0 0) 0x7fcfa01317f0 con 0x7fcfa01430c0 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:24:39.412 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.410+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa0136c50 con 0x7fcfa01430c0 2024-03-25T18:24:39.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1087374356 0 0) 0x7fcfa0136ec0 con 0x7fcfa00a98f0 2024-03-25T18:24:39.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfa01317f0 con 0x7fcfa00a98f0 2024-03-25T18:24:39.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcf900019b0 con 0x7fcfa00a98f0 2024-03-25T18:24:39.419 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 10 monclient(hunting): dump: 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2438059452 0 0) 0x7fcf8c004eb0 con 0x7fcfa013f710 2024-03-25T18:24:39.420 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfa0136ec0 con 0x7fcfa013f710 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcf940018b0 con 0x7fcfa013f710 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.418+0000 7fcf9cff9640 10 monclient(hunting): dump: 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:39.421 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:39.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3527353059 0 0) 0x7fcfa0136ec0 con 0x7fcfa013f710 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 >> v1:172.21.15.162:6790/0 conn(0x7fcfa01430c0 legacy=0x7fcfa01db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 >> v1:172.21.15.162:6789/0 conn(0x7fcfa00a98f0 legacy=0x7fcfa01310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 1 monclient: found mon.b 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa0131980 con 0x7fcfa013f710 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcf9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 10 monclient: _renew_subs 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa0135070 con 0x7fcfa013f710 2024-03-25T18:24:39.435 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:39.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 10 monclient: _renew_subs 2024-03-25T18:24:39.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:39.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa00b7870 con 0x7fcfa013f710 2024-03-25T18:24:39.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.433+0000 7fcfa68b5640 1 librados: init done 2024-03-25T18:24:39.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.434+0000 7fcfa68b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:39.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.434+0000 7fcfa68b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:39.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.434+0000 7fcfa68b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:39.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.434+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa00627b0 con 0x7fcfa013f710 2024-03-25T18:24:39.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.721+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf94003610 con 0x7fcfa013f710 2024-03-25T18:24:39.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.721+0000 7fcf9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:39.722 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.721+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fcf94003920 con 0x7fcfa013f710 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.721+0000 7fcf9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.721+0000 7fcf9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.721+0000 7fcf9cff9640 10 monclient: dump: 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.721+0000 7fcf9e7fc640 4 set_mon_vals no callback set 2024-03-25T18:24:39.723 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.721+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fcf940198f0 con 0x7fcfa013f710 2024-03-25T18:24:39.724 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.722+0000 7fcf9ffff640 10 monclient: get_auth_request con 0x7fcf8c06df70 auth_method 0 2024-03-25T18:24:39.724 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.722+0000 7fcf9cff9640 10 client.4693.objecter ms_handle_connect 0x7fcf8c06df70 2024-03-25T18:24:39.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.727+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7fcf940535a0 con 0x7fcfa013f710 2024-03-25T18:24:39.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.727+0000 7fcf9cff9640 10 client.4693.objecter ms_dispatch 0x7fcfa0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:39.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.727+0000 7fcf9cff9640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:39.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.727+0000 7fcf9cff9640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:39.728 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.727+0000 7fcf9cff9640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T18:24:39.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.728+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf9404eee0 con 0x7fcfa013f710 2024-03-25T18:24:39.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.728+0000 7fcf9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:39.729 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.728+0000 7fcf9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:39.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.954+0000 7fcfa68b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:39.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.954+0000 7fcfa68b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:39.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.954+0000 7fcfa68b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:39.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.954+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfa00b7ac0 con 0x7fcfa013f710 2024-03-25T18:24:39.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.987+0000 7fcf9cff9640 1 -- 172.21.15.162:0/3988655305 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7fcf94050fe0 con 0x7fcfa013f710 2024-03-25T18:24:39.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.987+0000 7fcf9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:39.989 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.987+0000 7fcf9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:39.989 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:24:39.989 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:39.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 10 librados: watch_flush enter 2024-03-25T18:24:39.991 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 10 librados: watch_flush exit 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 10 client.4693.objecter successfully canceled tick 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 >> v1:172.21.15.191:6800/1552703795 conn(0x7fcf8c06df70 legacy=0x7fcf8c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 10 monclient: shutdown 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 >> v1:172.21.15.191:6789/0 conn(0x7fcfa013f710 legacy=0x7fcfa01dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 shutdown_connections 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 >> 172.21.15.162:0/3988655305 conn(0x7fcfa00b7ea0 msgr2=0x7fcfa013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 shutdown_connections 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 1 -- 172.21.15.162:0/3988655305 wait complete. 2024-03-25T18:24:39.992 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:39.990+0000 7fcfa68b5640 1 librados: shutdown 2024-03-25T18:24:40.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:44.058 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:56.286 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.063 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:56.290 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.096 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:10:56.324 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.119 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:56.335 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.124 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:56.340 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.127 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:56.354 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.133 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:56.361 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.159 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:10:56.375 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.160 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:10:56.388 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:46.809 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.921+0000 7f9ab350e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.921+0000 7f9ab350e640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.921+0000 7f9ab350e640 10 monclient: monmap: 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:24:46.922582+0000 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:24:46.922582+0000 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:24:46.923 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:24:46.924 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:46.924 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.922+0000 7f9ab350e640 1 Processor -- start 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.922+0000 7f9ab350e640 1 -- start start 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.922+0000 7f9ab350e640 10 monclient: init 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 10 monclient(hunting): picked mon.noname-c con 0x7f9aac0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 10 monclient(hunting): picked mon.noname-a con 0x7f9aac0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 10 monclient(hunting): picked mon.noname-b con 0x7f9aac14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0af940 con 0x7f9aac0b1730 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0b0a70 con 0x7f9aac0b4d60 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0b0d50 con 0x7f9aac14be90 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.925 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.923+0000 7f9ab350e640 10 monclient(hunting): authenticate will time out at 8186.196825s 2024-03-25T18:24:46.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.934+0000 7f9ab1aab640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f9aac14be90 0x7f9aac14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:43644/0 (socket says 172.21.15.162:43644) 2024-03-25T18:24:46.935 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.934+0000 7f9ab1aab640 1 -- 172.21.15.162:0/1641666905 learned_addr learned my addr 172.21.15.162:0/1641666905 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:46.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.943+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590923997 0 0) 0x7f9aac0b0d50 con 0x7f9aac14be90 2024-03-25T18:24:46.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.943+0000 7f9aa3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.943+0000 7f9aa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.943+0000 7f9aa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.943+0000 7f9aa3fff640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:24:46.944 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.943+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a98003520 con 0x7f9aac14be90 2024-03-25T18:24:46.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.947+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771553981 0 0) 0x7f9aac0af940 con 0x7f9aac0b1730 2024-03-25T18:24:46.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.947+0000 7f9aa3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.947+0000 7f9aa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.947+0000 7f9aa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.947+0000 7f9aa3fff640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:24:46.948 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.947+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac0b0d50 con 0x7f9aac0b1730 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.955+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397060657 0 0) 0x7f9aac0b0a70 con 0x7f9aac0b4d60 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.955+0000 7f9aa3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.955+0000 7f9aa3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.955+0000 7f9aa3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.955+0000 7f9aa3fff640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:24:46.956 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.955+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac0af940 con 0x7f9aac0b4d60 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1426648260 0 0) 0x7f9aac0b0d50 con 0x7f9aac0b1730 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 >> v1:172.21.15.162:6790/0 conn(0x7f9aac14be90 legacy=0x7f9aac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 >> v1:172.21.15.191:6789/0 conn(0x7f9aac0b4d60 legacy=0x7f9aac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 1 monclient: found mon.noname-a 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac0b1030 con 0x7f9aac0b1730 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:46.958 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9a9c001670 con 0x7f9aac0b1730 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.956+0000 7f9aa3fff640 10 monclient: dump: 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:46.959 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:46.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.957+0000 7f9ab350e640 5 monclient: authenticate success, global_id 4719 2024-03-25T18:24:46.960 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.957+0000 7f9ab350e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:46.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.963+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a9c0017d0 con 0x7f9aac0b1730 2024-03-25T18:24:46.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.963+0000 7f9aa3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:46.964 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.963+0000 7f9aa3fff640 1 -- 172.21.15.162:0/1641666905 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9a9c001be0 con 0x7f9aac0b1730 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.963+0000 7f9ab350e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.963+0000 7f9ab350e640 4 set_mon_vals no callback set 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.963+0000 7f9aa3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.963+0000 7f9aa3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.963+0000 7f9aa3fff640 10 monclient: dump: 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:46.965 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.964+0000 7f9ab350e640 10 monclient: shutdown 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.964+0000 7f9ab350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.964+0000 7f9ab350e640 1 -- 172.21.15.162:0/1641666905 >> v1:172.21.15.162:6789/0 conn(0x7f9aac0b1730 legacy=0x7f9aac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.964+0000 7f9ab350e640 1 -- 172.21.15.162:0/1641666905 shutdown_connections 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.964+0000 7f9ab350e640 1 -- 172.21.15.162:0/1641666905 >> 172.21.15.162:0/1641666905 conn(0x7f9aac0617a0 msgr2=0x7f9aac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.964+0000 7f9ab350e640 1 -- 172.21.15.162:0/1641666905 shutdown_connections 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.964+0000 7f9ab350e640 1 -- 172.21.15.162:0/1641666905 wait complete. 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 10 monclient: build_initial_monmap 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 10 monclient: monmap: 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:46.968 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 1 librados: starting msgr at 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 1 librados: starting objecter 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 1 Processor -- start 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 1 -- start start 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 1 librados: setting wanted keys 2024-03-25T18:24:46.969 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 1 librados: calling monclient init 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.965+0000 7f9ab350e640 10 monclient: init 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 10 monclient(hunting): picked mon.noname-a con 0x7f9aac0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 10 monclient(hunting): picked mon.noname-b con 0x7f9aac0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 10 monclient(hunting): picked mon.noname-c con 0x7f9aac14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac1d79c0 con 0x7f9aac0b1730 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac1d7690 con 0x7f9aac0b4d60 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac1d8560 con 0x7f9aac14be90 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:46.970 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.966+0000 7f9ab350e640 10 monclient(hunting): authenticate will time out at 8186.240572s 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.976+0000 7f9ab12aa640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7f9aac0b1730 0x7f9aac1d7e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:54426/0 (socket says 172.21.15.162:54426) 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.976+0000 7f9ab12aa640 1 -- 172.21.15.162:0/2482096118 learned_addr learned my addr 172.21.15.162:0/2482096118 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.977+0000 7f9aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f9aac0b1730 2024-03-25T18:24:46.978 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.977+0000 7f9aa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.983 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.982+0000 7f9aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f9aac0b4d60 2024-03-25T18:24:46.983 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.982+0000 7f9aa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.985 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.984+0000 7f9aa1ffb640 10 client.?.objecter ms_handle_connect 0x7f9aac14be90 2024-03-25T18:24:46.985 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.984+0000 7f9aa1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:46.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.986+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408456462 0 0) 0x7f9aac1d7690 con 0x7f9aac0b4d60 2024-03-25T18:24:46.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.987+0000 7f9aa1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:46.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.987+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:46.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.987+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:46.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.987+0000 7f9aa1ffb640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:24:46.988 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.987+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a90004eb0 con 0x7f9aac0b4d60 2024-03-25T18:24:46.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.997+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2280407863 0 0) 0x7f9a90004eb0 con 0x7f9aac0b4d60 2024-03-25T18:24:46.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9aac1d7690 con 0x7f9aac0b4d60 2024-03-25T18:24:46.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9a9c001b20 con 0x7f9aac0b4d60 2024-03-25T18:24:46.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:46.999 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221758312 0 0) 0x7f9aac1d8560 con 0x7f9aac14be90 2024-03-25T18:24:47.000 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a90004eb0 con 0x7f9aac14be90 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448253292 0 0) 0x7f9aac1d79c0 con 0x7f9aac0b1730 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:24:47.001 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:46.998+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac1d8560 con 0x7f9aac0b1730 2024-03-25T18:24:47.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1715068575 0 0) 0x7f9aac1d7690 con 0x7f9aac0b4d60 2024-03-25T18:24:47.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 >> v1:172.21.15.162:6790/0 conn(0x7f9aac14be90 legacy=0x7f9aac1e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 >> v1:172.21.15.162:6789/0 conn(0x7f9aac0b1730 legacy=0x7f9aac1d7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:47.008 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 1 monclient: found mon.b 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac142b40 con 0x7f9aac0b4d60 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9aa1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9ab350e640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9ab350e640 10 monclient: _renew_subs 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9ab350e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.006+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac142d90 con 0x7f9aac0b4d60 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.007+0000 7f9ab350e640 10 client.4699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.007+0000 7f9ab350e640 10 monclient: _renew_subs 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.007+0000 7f9ab350e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:47.009 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.007+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac142860 con 0x7f9aac0b4d60 2024-03-25T18:24:47.010 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.007+0000 7f9ab350e640 1 librados: init done 2024-03-25T18:24:47.010 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.008+0000 7f9ab350e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.010 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.008+0000 7f9ab350e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.010 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.008+0000 7f9ab350e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:47.010 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.008+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aac0aacb0 con 0x7f9aac0b4d60 2024-03-25T18:24:47.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.283+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a9c0038a0 con 0x7f9aac0b4d60 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.283+0000 7f9aa1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.283+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f9a9c003bb0 con 0x7f9aac0b4d60 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.283+0000 7f9aa1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.283+0000 7f9aa1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.283+0000 7f9aa1ffb640 10 monclient: dump: 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:47.285 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:47.286 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:47.286 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:47.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.283+0000 7f9aa37fe640 4 set_mon_vals no callback set 2024-03-25T18:24:47.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.284+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f9a9c019b80 con 0x7f9aac0b4d60 2024-03-25T18:24:47.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.285+0000 7f9ab12aa640 10 monclient: get_auth_request con 0x7f9a9006dea0 auth_method 0 2024-03-25T18:24:47.286 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.285+0000 7f9aa1ffb640 10 client.4699.objecter ms_handle_connect 0x7f9a9006dea0 2024-03-25T18:24:47.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.290+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7f9a9c053720 con 0x7f9aac0b4d60 2024-03-25T18:24:47.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.290+0000 7f9aa1ffb640 10 client.4699.objecter ms_dispatch 0x7f9aac000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:47.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.290+0000 7f9aa1ffb640 3 client.4699.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:47.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.290+0000 7f9aa1ffb640 3 client.4699.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:47.292 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.291+0000 7f9aa1ffb640 20 client.4699.objecter dump_active .. 0 homeless 2024-03-25T18:24:47.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.292+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a9c04f230 con 0x7f9aac0b4d60 2024-03-25T18:24:47.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.292+0000 7f9aa1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:47.293 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.292+0000 7f9aa1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:47.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.541+0000 7f9ab350e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:47.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.541+0000 7f9ab350e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:47.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.541+0000 7f9ab350e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:47.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.541+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9aac1d7b70 con 0x7f9aac0b4d60 2024-03-25T18:24:47.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.562+0000 7f9aa1ffb640 1 -- 172.21.15.162:0/2482096118 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7f9aac1d7b70 con 0x7f9aac0b4d60 2024-03-25T18:24:47.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.562+0000 7f9aa1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:47.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.562+0000 7f9aa1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:47.563 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:24:47.563 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:47.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 10 librados: watch_flush enter 2024-03-25T18:24:47.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 10 librados: watch_flush exit 2024-03-25T18:24:47.566 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 20 client.4699.objecter shutdown clearing up homeless session... 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 10 client.4699.objecter successfully canceled tick 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 >> v1:172.21.15.191:6800/1552703795 conn(0x7f9a9006dea0 legacy=0x7f9a90070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 10 monclient: shutdown 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 >> v1:172.21.15.191:6789/0 conn(0x7f9aac0b4d60 legacy=0x7f9aac137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 shutdown_connections 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 >> 172.21.15.162:0/2482096118 conn(0x7f9aac0617a0 msgr2=0x7f9aac1453a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 shutdown_connections 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 1 -- 172.21.15.162:0/2482096118 wait complete. 2024-03-25T18:24:47.567 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:47.565+0000 7f9ab350e640 1 librados: shutdown 2024-03-25T18:24:47.579 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:54.385 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:54.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.502+0000 7f8cf1104640 10 monclient: get_monmap_and_config 2024-03-25T18:24:54.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.502+0000 7f8cf1104640 10 monclient: build_initial_monmap 2024-03-25T18:24:54.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.502+0000 7f8cf1104640 10 monclient: monmap: 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:24:54.503223+0000 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:24:54.503223+0000 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:24:54.504 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:54.505 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 1 Processor -- start 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 1 -- start start 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 10 monclient: init 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 10 monclient(hunting): picked mon.noname-a con 0x7f8cec0b4d60 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 10 monclient(hunting): picked mon.noname-b con 0x7f8cec0b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 10 monclient(hunting): picked mon.noname-c con 0x7f8cec14be90 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0af940 con 0x7f8cec0b4d60 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0b0a70 con 0x7f8cec14be90 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec0b0d50 con 0x7f8cec0b1730 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:54.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.503+0000 7f8cf1104640 10 monclient(hunting): authenticate will time out at 8193.777521s 2024-03-25T18:24:54.511 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.510+0000 7f8ce3fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f8cec0b1730 0x7f8cec0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:57470/0 (socket says 172.21.15.162:57470) 2024-03-25T18:24:54.511 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.510+0000 7f8ce3fff640 1 -- 172.21.15.162:0/4199084604 learned_addr learned my addr 172.21.15.162:0/4199084604 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:54.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.524+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890017832 0 0) 0x7f8cec0b0a70 con 0x7f8cec14be90 2024-03-25T18:24:54.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.524+0000 7f8cea59c640 10 monclient(hunting): _negotiate 2024-03-25T18:24:54.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.524+0000 7f8cea59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:54.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.524+0000 7f8cea59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:54.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.524+0000 7f8cea59c640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:24:54.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.525+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd8003520 con 0x7f8cec14be90 2024-03-25T18:24:54.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2947022771 0 0) 0x7f8cd8003520 con 0x7f8cec14be90 2024-03-25T18:24:54.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 >> v1:172.21.15.162:6790/0 conn(0x7f8cec0b1730 legacy=0x7f8cec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 >> v1:172.21.15.162:6789/0 conn(0x7f8cec0b4d60 legacy=0x7f8cec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:54.529 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 10 monclient: _finish_hunting 0 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 1 monclient: found mon.noname-c 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec0b1030 con 0x7f8cec14be90 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 10 monclient: _finish_auth 0 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 10 monclient: _check_auth_tickets 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8ce4001a10 con 0x7f8cec14be90 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cea59c640 10 monclient: dump: 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:54.530 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cf1104640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:24:54.531 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.528+0000 7f8cf1104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.537+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ce4001be0 con 0x7f8cec14be90 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.537+0000 7f8cea59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.537+0000 7f8cea59c640 1 -- 172.21.15.162:0/4199084604 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8ce4002030 con 0x7f8cec14be90 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.537+0000 7f8cea59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.537+0000 7f8cea59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.537+0000 7f8cea59c640 10 monclient: dump: 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:54.539 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:54.540 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:54.540 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:54.540 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:54.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.538+0000 7f8cf1104640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:54.540 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.538+0000 7f8cf1104640 4 set_mon_vals no callback set 2024-03-25T18:24:54.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.538+0000 7f8cf1104640 10 monclient: shutdown 2024-03-25T18:24:54.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.538+0000 7f8cf1104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:54.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.538+0000 7f8cf1104640 1 -- 172.21.15.162:0/4199084604 >> v1:172.21.15.191:6789/0 conn(0x7f8cec14be90 legacy=0x7f8cec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.538+0000 7f8cf1104640 1 -- 172.21.15.162:0/4199084604 shutdown_connections 2024-03-25T18:24:54.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.538+0000 7f8cf1104640 1 -- 172.21.15.162:0/4199084604 >> 172.21.15.162:0/4199084604 conn(0x7f8cec0617a0 msgr2=0x7f8cec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:54.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.539+0000 7f8cf1104640 1 -- 172.21.15.162:0/4199084604 shutdown_connections 2024-03-25T18:24:54.541 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.539+0000 7f8cf1104640 1 -- 172.21.15.162:0/4199084604 wait complete. 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 10 monclient: build_initial_monmap 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 10 monclient: monmap: 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 1 librados: starting msgr at 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 1 librados: starting objecter 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 1 Processor -- start 2024-03-25T18:24:54.542 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 1 -- start start 2024-03-25T18:24:54.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 1 librados: setting wanted keys 2024-03-25T18:24:54.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 1 librados: calling monclient init 2024-03-25T18:24:54.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.540+0000 7f8cf1104640 10 monclient: init 2024-03-25T18:24:54.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:54.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:54.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 10 monclient(hunting): picked mon.noname-a con 0x7f8cec0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:24:54.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 10 monclient(hunting): picked mon.noname-b con 0x7f8cec0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:24:54.543 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 10 monclient(hunting): picked mon.noname-c con 0x7f8cec14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:24:54.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec1d7a40 con 0x7f8cec0b1730 2024-03-25T18:24:54.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec13a9d0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cec1d85e0 con 0x7f8cec14be90 2024-03-25T18:24:54.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:54.544 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.541+0000 7f8cf1104640 10 monclient(hunting): authenticate will time out at 8193.814914s 2024-03-25T18:24:54.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.549+0000 7f8ce3fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8cec0b4d60 0x7f8cec1d7ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:49228/0 (socket says 172.21.15.162:49228) 2024-03-25T18:24:54.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.549+0000 7f8ce3fff640 1 -- 172.21.15.162:0/230215031 learned_addr learned my addr 172.21.15.162:0/230215031 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:24:54.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.549+0000 7f8ce37fe640 10 client.?.objecter ms_handle_connect 0x7f8cec0b4d60 2024-03-25T18:24:54.550 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.549+0000 7f8ce37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:54.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.553+0000 7f8ce37fe640 10 client.?.objecter ms_handle_connect 0x7f8cec14be90 2024-03-25T18:24:54.554 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.553+0000 7f8ce37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:54.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.562+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937921223 0 0) 0x7f8cec13a9d0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.562+0000 7f8ce37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:54.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.562+0000 7f8ce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:54.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.562+0000 7f8ce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:54.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.562+0000 7f8ce37fe640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:24:54.563 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.562+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ccc004eb0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.568+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370595283 0 0) 0x7f8cec1d85e0 con 0x7f8cec14be90 2024-03-25T18:24:54.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.568+0000 7f8ce37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:54.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.568+0000 7f8ce37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:54.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.568+0000 7f8ce37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:54.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.568+0000 7f8ce37fe640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:24:54.569 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.568+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cec13a9d0 con 0x7f8cec14be90 2024-03-25T18:24:54.576 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.575+0000 7f8ce37fe640 10 client.?.objecter ms_handle_connect 0x7f8cec0b1730 2024-03-25T18:24:54.576 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.575+0000 7f8ce37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:54.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.578+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 229536399 0 0) 0x7f8cec13a9d0 con 0x7f8cec14be90 2024-03-25T18:24:54.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.578+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cec1d85e0 con 0x7f8cec14be90 2024-03-25T18:24:54.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.578+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8ce4001ef0 con 0x7f8cec14be90 2024-03-25T18:24:54.579 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.578+0000 7f8ce37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.578+0000 7f8ce37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.578+0000 7f8ce37fe640 10 monclient(hunting): dump: 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:54.580 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.578+0000 7f8ce37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.582+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4190932115 0 0) 0x7f8ccc004eb0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.582+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cec13a9d0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.582+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8cd4001930 con 0x7f8cec0b4d60 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.582+0000 7f8ce37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.582+0000 7f8ce37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.582+0000 7f8ce37fe640 10 monclient(hunting): dump: 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:54.583 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:54.584 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:54.584 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:54.584 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:54.596 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3033217037 0 0) 0x7f8cec13a9d0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 >> v1:172.21.15.162:6790/0 conn(0x7f8cec14be90 legacy=0x7f8cec13b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 >> v1:172.21.15.162:6789/0 conn(0x7f8cec0b1730 legacy=0x7f8cec139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 1 monclient: found mon.b 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cec13bf30 con 0x7f8cec0b4d60 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8ce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 10 monclient: _renew_subs 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:54.597 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec13c1e0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:54.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 10 monclient: _renew_subs 2024-03-25T18:24:54.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:54.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec142b90 con 0x7f8cec0b4d60 2024-03-25T18:24:54.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.595+0000 7f8cf1104640 1 librados: init done 2024-03-25T18:24:54.598 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.596+0000 7f8cf1104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:54.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.596+0000 7f8cf1104640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:54.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.596+0000 7f8cf1104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:54.599 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.597+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec1d77a0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.870+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cd40047e0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.870+0000 7f8ce37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:54.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.870+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f8cd4004bf0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.871 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.870+0000 7f8ce37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.870+0000 7f8ce37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.870+0000 7f8ce37fe640 10 monclient: dump: 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:24:54.872 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.870+0000 7f8ce9d9b640 4 set_mon_vals no callback set 2024-03-25T18:24:54.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.871+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f8cd4004e10 con 0x7f8cec0b4d60 2024-03-25T18:24:54.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.871+0000 7f8cead9d640 10 monclient: get_auth_request con 0x7f8ccc06de50 auth_method 0 2024-03-25T18:24:54.873 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.872+0000 7f8ce37fe640 10 client.4705.objecter ms_handle_connect 0x7f8ccc06de50 2024-03-25T18:24:54.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.876+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7f8cd4054890 con 0x7f8cec0b4d60 2024-03-25T18:24:54.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.877+0000 7f8ce37fe640 10 client.4705.objecter ms_dispatch 0x7f8cec000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:54.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.877+0000 7f8ce37fe640 3 client.4705.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:54.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.877+0000 7f8ce37fe640 3 client.4705.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:54.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.877+0000 7f8ce37fe640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T18:24:54.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.877+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8cd40503a0 con 0x7f8cec0b4d60 2024-03-25T18:24:54.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.877+0000 7f8ce37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:54.878 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:54.877+0000 7f8ce37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:55.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.116+0000 7f8cf1104640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:55.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.116+0000 7f8cf1104640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:55.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.116+0000 7f8cf1104640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:24:55.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.116+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8cec1d7bf0 con 0x7f8cec0b4d60 2024-03-25T18:24:55.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.142+0000 7f8ce37fe640 1 -- 172.21.15.162:0/230215031 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7f8cec1d7bf0 con 0x7f8cec0b4d60 2024-03-25T18:24:55.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.142+0000 7f8ce37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:55.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.142+0000 7f8ce37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:55.143 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:24:55.143 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 10 librados: watch_flush enter 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 10 librados: watch_flush exit 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 10 client.4705.objecter successfully canceled tick 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 >> v1:172.21.15.191:6800/1552703795 conn(0x7f8ccc06de50 legacy=0x7f8ccc070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 10 monclient: shutdown 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 >> v1:172.21.15.191:6789/0 conn(0x7f8cec0b4d60 legacy=0x7f8cec1d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 shutdown_connections 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 >> 172.21.15.162:0/230215031 conn(0x7f8cec0617a0 msgr2=0x7f8cec137120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:55.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 shutdown_connections 2024-03-25T18:24:55.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.144+0000 7f8cf1104640 1 -- 172.21.15.162:0/230215031 wait complete. 2024-03-25T18:24:55.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:24:55.145+0000 7f8cf1104640 1 librados: shutdown 2024-03-25T18:24:55.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:01.966 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:02.082 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.081+0000 7f1f3eb32640 10 monclient: get_monmap_and_config 2024-03-25T18:25:02.082 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.081+0000 7f1f3eb32640 10 monclient: build_initial_monmap 2024-03-25T18:25:02.082 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.081+0000 7f1f3eb32640 10 monclient: monmap: 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:25:02.082519+0000 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:25:02.082519+0000 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:25:02.083 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:02.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.082+0000 7f1f3eb32640 1 Processor -- start 2024-03-25T18:25:02.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.082+0000 7f1f3eb32640 1 -- start start 2024-03-25T18:25:02.084 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.082+0000 7f1f3eb32640 10 monclient: init 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 10 monclient(hunting): picked mon.noname-c con 0x7f1f380b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 10 monclient(hunting): picked mon.noname-b con 0x7f1f380b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 10 monclient(hunting): picked mon.noname-a con 0x7f1f3814be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380af940 con 0x7f1f3814be90 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380b0a70 con 0x7f1f380b4d60 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f380b0d50 con 0x7f1f380b1730 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:02.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.083+0000 7f1f3eb32640 10 monclient(hunting): authenticate will time out at 8201.357625s 2024-03-25T18:25:02.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.089+0000 7f1f37fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f1f380b1730 0x7f1f380b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:34018/0 (socket says 172.21.15.162:34018) 2024-03-25T18:25:02.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.089+0000 7f1f37fff640 1 -- 172.21.15.162:0/2501914245 learned_addr learned my addr 172.21.15.162:0/2501914245 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:02.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.106+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041762053 0 0) 0x7f1f380b0d50 con 0x7f1f380b1730 2024-03-25T18:25:02.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.106+0000 7f1f377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:02.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.106+0000 7f1f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:02.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.106+0000 7f1f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:02.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.106+0000 7f1f377fe640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:25:02.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.106+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f28003520 con 0x7f1f380b1730 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077410889 0 0) 0x7f1f380af940 con 0x7f1f3814be90 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f380b0d50 con 0x7f1f3814be90 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640140878 0 0) 0x7f1f380b0a70 con 0x7f1f380b4d60 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:02.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:02.116 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:02.116 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:25:02.116 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.114+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f380af940 con 0x7f1f380b4d60 2024-03-25T18:25:02.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 76681804 0 0) 0x7f1f380af940 con 0x7f1f380b4d60 2024-03-25T18:25:02.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 >> v1:172.21.15.162:6790/0 conn(0x7f1f380b1730 legacy=0x7f1f380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 >> v1:172.21.15.162:6789/0 conn(0x7f1f3814be90 legacy=0x7f1f3814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 1 monclient: found mon.noname-c 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f380b1030 con 0x7f1f380b4d60 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1f2c0015b0 con 0x7f1f380b4d60 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f3eb32640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f3eb32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:02.125 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f377fe640 10 monclient: dump: 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:02.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.123+0000 7f1f3eb32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:02.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.142+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f2c001710 con 0x7f1f380b4d60 2024-03-25T18:25:02.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.142+0000 7f1f377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:02.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.142+0000 7f1f377fe640 1 -- 172.21.15.162:0/2501914245 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1f2c001b60 con 0x7f1f380b4d60 2024-03-25T18:25:02.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.142+0000 7f1f3eb32640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.142+0000 7f1f3eb32640 4 set_mon_vals no callback set 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.142+0000 7f1f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.142+0000 7f1f377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.142+0000 7f1f377fe640 10 monclient: dump: 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:02.144 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.143+0000 7f1f3eb32640 10 monclient: shutdown 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.143+0000 7f1f3eb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.143+0000 7f1f3eb32640 1 -- 172.21.15.162:0/2501914245 >> v1:172.21.15.191:6789/0 conn(0x7f1f380b4d60 legacy=0x7f1f3813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.143+0000 7f1f3eb32640 1 -- 172.21.15.162:0/2501914245 shutdown_connections 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.143+0000 7f1f3eb32640 1 -- 172.21.15.162:0/2501914245 >> 172.21.15.162:0/2501914245 conn(0x7f1f380617a0 msgr2=0x7f1f38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.143+0000 7f1f3eb32640 1 -- 172.21.15.162:0/2501914245 shutdown_connections 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.143+0000 7f1f3eb32640 1 -- 172.21.15.162:0/2501914245 wait complete. 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.144+0000 7f1f3eb32640 10 monclient: build_initial_monmap 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.144+0000 7f1f3eb32640 10 monclient: monmap: 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:02.146 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.144+0000 7f1f3eb32640 1 librados: starting msgr at 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.144+0000 7f1f3eb32640 1 librados: starting objecter 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.144+0000 7f1f3eb32640 1 Processor -- start 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 1 -- start start 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 1 librados: setting wanted keys 2024-03-25T18:25:02.147 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 1 librados: calling monclient init 2024-03-25T18:25:02.148 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 10 monclient: init 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 10 monclient(hunting): picked mon.noname-a con 0x7f1f380b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 10 monclient(hunting): picked mon.noname-b con 0x7f1f380b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 10 monclient(hunting): picked mon.noname-c con 0x7f1f3814be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f381d79d0 con 0x7f1f380b1730 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f381d8570 con 0x7f1f380b4d60 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f38142680 con 0x7f1f3814be90 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:02.149 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.145+0000 7f1f3eb32640 10 monclient(hunting): authenticate will time out at 8201.419608s 2024-03-25T18:25:02.161 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.160+0000 7f1f37fff640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1f380b4d60 0x7f1f38137d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:53798/0 (socket says 172.21.15.162:53798) 2024-03-25T18:25:02.161 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.160+0000 7f1f37fff640 1 -- 172.21.15.162:0/1950626077 learned_addr learned my addr 172.21.15.162:0/1950626077 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:02.163 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.162+0000 7f1f357fa640 10 client.?.objecter ms_handle_connect 0x7f1f380b4d60 2024-03-25T18:25:02.163 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.162+0000 7f1f357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:02.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.168+0000 7f1f357fa640 10 client.?.objecter ms_handle_connect 0x7f1f3814be90 2024-03-25T18:25:02.170 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.168+0000 7f1f357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:02.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.180+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356704981 0 0) 0x7f1f381d8570 con 0x7f1f380b4d60 2024-03-25T18:25:02.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.180+0000 7f1f357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:02.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.180+0000 7f1f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:02.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.180+0000 7f1f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:02.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.180+0000 7f1f357fa640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:25:02.181 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.180+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f1c004eb0 con 0x7f1f380b4d60 2024-03-25T18:25:02.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.183+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029893274 0 0) 0x7f1f38142680 con 0x7f1f3814be90 2024-03-25T18:25:02.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.183+0000 7f1f357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:02.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.183+0000 7f1f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:02.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.183+0000 7f1f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:02.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.183+0000 7f1f357fa640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:25:02.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.183+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f381d8570 con 0x7f1f3814be90 2024-03-25T18:25:02.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.183+0000 7f1f357fa640 10 client.?.objecter ms_handle_connect 0x7f1f380b1730 2024-03-25T18:25:02.184 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.183+0000 7f1f357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.200+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 255631353 0 0) 0x7f1f381d8570 con 0x7f1f3814be90 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.200+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f38142680 con 0x7f1f3814be90 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.200+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1f30001bd0 con 0x7f1f3814be90 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.200+0000 7f1f357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.200+0000 7f1f357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.200+0000 7f1f357fa640 10 monclient(hunting): dump: 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.202 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:02.203 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:02.203 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:02.203 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:02.203 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:02.203 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:02.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.200+0000 7f1f357fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:02.203 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.202+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512173004 0 0) 0x7f1f381d79d0 con 0x7f1f380b1730 2024-03-25T18:25:02.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.202+0000 7f1f357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:02.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.202+0000 7f1f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:02.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.202+0000 7f1f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:02.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.202+0000 7f1f357fa640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:25:02.204 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.202+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f381d8570 con 0x7f1f380b1730 2024-03-25T18:25:02.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.210+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2890194211 0 0) 0x7f1f1c004eb0 con 0x7f1f380b4d60 2024-03-25T18:25:02.211 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.210+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f381d79d0 con 0x7f1f380b4d60 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.210+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1f24001950 con 0x7f1f380b4d60 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.210+0000 7f1f357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.210+0000 7f1f357fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.210+0000 7f1f357fa640 10 monclient(hunting): dump: 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:02.212 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:02.216 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2242211660 0 0) 0x7f1f38142680 con 0x7f1f3814be90 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 >> v1:172.21.15.191:6789/0 conn(0x7f1f380b4d60 legacy=0x7f1f38137d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 >> v1:172.21.15.162:6789/0 conn(0x7f1f380b1730 legacy=0x7f1f381d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 1 monclient: found mon.c 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f38142810 con 0x7f1f3814be90 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 10 monclient: _renew_subs 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:02.217 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f38142b60 con 0x7f1f3814be90 2024-03-25T18:25:02.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:02.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 10 monclient: _renew_subs 2024-03-25T18:25:02.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:02.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f380a98c0 con 0x7f1f3814be90 2024-03-25T18:25:02.218 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.215+0000 7f1f3eb32640 1 librados: init done 2024-03-25T18:25:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.216+0000 7f1f3eb32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.216+0000 7f1f3eb32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.216+0000 7f1f3eb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:02.219 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.216+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f380aad20 con 0x7f1f3814be90 2024-03-25T18:25:02.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.536+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f30002080 con 0x7f1f3814be90 2024-03-25T18:25:02.537 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.536+0000 7f1f357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.536+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f1f30003b30 con 0x7f1f3814be90 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.536+0000 7f1f357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.536+0000 7f1f357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.536+0000 7f1f357fa640 10 monclient: dump: 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:02.538 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:02.539 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:02.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.536+0000 7f1f36ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:02.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.537+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f1f30019b40 con 0x7f1f3814be90 2024-03-25T18:25:02.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.538+0000 7f1f3c8ce640 10 monclient: get_auth_request con 0x7f1f1c06df70 auth_method 0 2024-03-25T18:25:02.539 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.538+0000 7f1f357fa640 10 client.4715.objecter ms_handle_connect 0x7f1f1c06df70 2024-03-25T18:25:02.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.545+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7f1f300537d0 con 0x7f1f3814be90 2024-03-25T18:25:02.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.545+0000 7f1f357fa640 10 client.4715.objecter ms_dispatch 0x7f1f38000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:02.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.545+0000 7f1f357fa640 3 client.4715.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:02.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.545+0000 7f1f357fa640 3 client.4715.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:02.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.545+0000 7f1f357fa640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-25T18:25:02.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.545+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f3004f110 con 0x7f1f3814be90 2024-03-25T18:25:02.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.546+0000 7f1f357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:02.547 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.546+0000 7f1f357fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:02.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.783+0000 7f1f3eb32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:02.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.783+0000 7f1f3eb32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:02.784 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.783+0000 7f1f3eb32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:02.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.783+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f381d7b80 con 0x7f1f3814be90 2024-03-25T18:25:02.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.848+0000 7f1f357fa640 1 -- 172.21.15.162:0/1950626077 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7f1f381d7b80 con 0x7f1f3814be90 2024-03-25T18:25:02.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.848+0000 7f1f357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:02.850 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.848+0000 7f1f357fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:02.850 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:25:02.850 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 10 librados: watch_flush enter 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 10 librados: watch_flush exit 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 10 client.4715.objecter successfully canceled tick 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 >> v1:172.21.15.191:6800/1552703795 conn(0x7f1f1c06df70 legacy=0x7f1f1c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 10 monclient: shutdown 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 >> v1:172.21.15.162:6790/0 conn(0x7f1f3814be90 legacy=0x7f1f381e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 shutdown_connections 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.850+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 >> 172.21.15.162:0/1950626077 conn(0x7f1f380617a0 msgr2=0x7f1f38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.851+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 shutdown_connections 2024-03-25T18:25:02.852 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.851+0000 7f1f3eb32640 1 -- 172.21.15.162:0/1950626077 wait complete. 2024-03-25T18:25:02.853 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:02.851+0000 7f1f3eb32640 1 librados: shutdown 2024-03-25T18:25:02.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:04.131 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:16.359 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:04.148 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:16.375 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:04.180 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:16.408 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:04.195 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:16.411 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:04.196 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:16.424 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:04.203 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:16.419 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:04.211 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:16.439 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:04.237 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:16.453 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:04.240 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:16.468 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.368 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:17.595 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.374 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:17.601 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.379 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:17.606 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.381 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:17.609 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.384 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:17.600 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.387 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:17.603 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.398 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:17.626 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.404 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:17.631 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:05.404 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:17.620 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:09.671 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.772+0000 7f2954e55640 10 monclient: get_monmap_and_config 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.772+0000 7f2954e55640 10 monclient: build_initial_monmap 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.773+0000 7f2954e55640 10 monclient: monmap: 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:25:09.774041+0000 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:25:09.774041+0000 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:09.774 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:09.775 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:09.775 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:25:09.775 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:25:09.775 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.773+0000 7f2954e55640 1 Processor -- start 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.773+0000 7f2954e55640 1 -- start start 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.773+0000 7f2954e55640 10 monclient: init 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 10 monclient(hunting): picked mon.noname-c con 0x7f29500b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 10 monclient(hunting): picked mon.noname-b con 0x7f29500b1730 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 10 monclient(hunting): picked mon.noname-a con 0x7f295014be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29500af940 con 0x7f295014be90 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29500b0a70 con 0x7f29500b4d60 2024-03-25T18:25:09.776 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29500b0d50 con 0x7f29500b1730 2024-03-25T18:25:09.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:09.777 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.774+0000 7f2954e55640 10 monclient(hunting): authenticate will time out at 8209.048022s 2024-03-25T18:25:09.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.784+0000 7f294e59c640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f29500b4d60 0x7f295013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:44402/0 (socket says 172.21.15.162:44402) 2024-03-25T18:25:09.785 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.784+0000 7f294e59c640 1 -- 172.21.15.162:0/2141940021 learned_addr learned my addr 172.21.15.162:0/2141940021 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:09.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.813+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501306618 0 0) 0x7f29500b0a70 con 0x7f29500b4d60 2024-03-25T18:25:09.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.813+0000 7f294d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.813+0000 7f294d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.814 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.813+0000 7f294d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.813+0000 7f294d59a640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:25:09.815 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.813+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2938003520 con 0x7f29500b4d60 2024-03-25T18:25:09.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.820+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456136262 0 0) 0x7f29500b0d50 con 0x7f29500b1730 2024-03-25T18:25:09.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.820+0000 7f294d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.820+0000 7f294d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.820+0000 7f294d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.820+0000 7f294d59a640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:25:09.821 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.820+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29500b0a70 con 0x7f29500b1730 2024-03-25T18:25:09.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.822+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739653966 0 0) 0x7f29500af940 con 0x7f295014be90 2024-03-25T18:25:09.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.822+0000 7f294d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.822+0000 7f294d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.822+0000 7f294d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.822+0000 7f294d59a640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:25:09.823 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.822+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29500b0d50 con 0x7f295014be90 2024-03-25T18:25:09.831 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178728114 0 0) 0x7f2938003520 con 0x7f29500b4d60 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 >> v1:172.21.15.162:6790/0 conn(0x7f29500b1730 legacy=0x7f29500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 >> v1:172.21.15.162:6789/0 conn(0x7f295014be90 legacy=0x7f295014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 1 monclient: found mon.noname-c 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.191:6789/0 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29500b1030 con 0x7f29500b4d60 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 10 monclient: _finish_auth 0 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2940001690 con 0x7f29500b4d60 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:09.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f294d59a640 10 monclient: dump: 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f2954e55640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:25:09.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.830+0000 7f2954e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:09.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.854+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 <== mon.1 v1:172.21.15.191:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29400017f0 con 0x7f29500b4d60 2024-03-25T18:25:09.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.854+0000 7f294d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:09.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.854+0000 7f294d59a640 1 -- 172.21.15.162:0/2141940021 <== mon.1 v1:172.21.15.191:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2940001c00 con 0x7f29500b4d60 2024-03-25T18:25:09.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.854+0000 7f294d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.854+0000 7f294d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:09.855 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.854+0000 7f294d59a640 10 monclient: dump: 2024-03-25T18:25:09.855 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:09.855 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.854+0000 7f2954e55640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:09.856 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.854+0000 7f2954e55640 4 set_mon_vals no callback set 2024-03-25T18:25:09.857 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.855+0000 7f2954e55640 10 monclient: shutdown 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.855+0000 7f2954e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.855+0000 7f2954e55640 1 -- 172.21.15.162:0/2141940021 >> v1:172.21.15.191:6789/0 conn(0x7f29500b4d60 legacy=0x7f295013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.855+0000 7f2954e55640 1 -- 172.21.15.162:0/2141940021 shutdown_connections 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.855+0000 7f2954e55640 1 -- 172.21.15.162:0/2141940021 >> 172.21.15.162:0/2141940021 conn(0x7f29500617a0 msgr2=0x7f2950061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.855+0000 7f2954e55640 1 -- 172.21.15.162:0/2141940021 shutdown_connections 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.855+0000 7f2954e55640 1 -- 172.21.15.162:0/2141940021 wait complete. 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.856+0000 7f2954e55640 10 monclient: build_initial_monmap 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.856+0000 7f2954e55640 10 monclient: monmap: 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:25:09.858 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.856+0000 7f2954e55640 1 librados: starting msgr at 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.856+0000 7f2954e55640 1 librados: starting objecter 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.856+0000 7f2954e55640 1 Processor -- start 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 1 -- start start 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 1 librados: setting wanted keys 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 1 librados: calling monclient init 2024-03-25T18:25:09.859 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 10 monclient: init 2024-03-25T18:25:09.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:09.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:09.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 10 monclient(hunting): picked mon.noname-b con 0x7f29500b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:09.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 10 monclient(hunting): picked mon.noname-a con 0x7f295014be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:09.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 10 monclient(hunting): picked mon.noname-c con 0x7f295013b8b0 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:09.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29501d77a0 con 0x7f295014be90 2024-03-25T18:25:09.860 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2950141d50 con 0x7f29500b1730 2024-03-25T18:25:09.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2950140d00 con 0x7f295013b8b0 2024-03-25T18:25:09.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:09.861 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.857+0000 7f2954e55640 10 monclient(hunting): authenticate will time out at 8209.131371s 2024-03-25T18:25:09.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.873+0000 7f294ed9d640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f295013b8b0 0x7f2950142c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:38676/0 (socket says 172.21.15.162:38676) 2024-03-25T18:25:09.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.873+0000 7f294ed9d640 1 -- 172.21.15.162:0/2483584831 learned_addr learned my addr 172.21.15.162:0/2483584831 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:09.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.873+0000 7f293effd640 10 client.?.objecter ms_handle_connect 0x7f295013b8b0 2024-03-25T18:25:09.874 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.873+0000 7f293effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.882+0000 7f293effd640 10 client.?.objecter ms_handle_connect 0x7f29500b1730 2024-03-25T18:25:09.883 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.882+0000 7f293effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.884+0000 7f293effd640 10 client.?.objecter ms_handle_connect 0x7f295014be90 2024-03-25T18:25:09.885 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.884+0000 7f293effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:09.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.890+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562755139 0 0) 0x7f2950141d50 con 0x7f29500b1730 2024-03-25T18:25:09.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.890+0000 7f293effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.890+0000 7f293effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.890+0000 7f293effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.890+0000 7f293effd640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:25:09.891 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.890+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2930004eb0 con 0x7f29500b1730 2024-03-25T18:25:09.892 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.891+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524434288 0 0) 0x7f29501d77a0 con 0x7f295014be90 2024-03-25T18:25:09.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.891+0000 7f293effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.891+0000 7f293effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.891+0000 7f293effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.891+0000 7f293effd640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:25:09.893 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.891+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2950141d50 con 0x7f295014be90 2024-03-25T18:25:09.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.898+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824397222 0 0) 0x7f2950140d00 con 0x7f295013b8b0 2024-03-25T18:25:09.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.898+0000 7f293effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:09.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.898+0000 7f293effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:09.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.898+0000 7f293effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:09.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.898+0000 7f293effd640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:25:09.899 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.898+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29501d77a0 con 0x7f295013b8b0 2024-03-25T18:25:09.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.908+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2210143459 0 0) 0x7f2950141d50 con 0x7f295014be90 2024-03-25T18:25:09.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.908+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2950140d00 con 0x7f295014be90 2024-03-25T18:25:09.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.908+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2944001970 con 0x7f295014be90 2024-03-25T18:25:09.909 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.908+0000 7f293effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.908+0000 7f293effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.908+0000 7f293effd640 10 monclient(hunting): dump: 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:09.910 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.908+0000 7f293effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:09.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.910+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1413088892 0 0) 0x7f2930004eb0 con 0x7f29500b1730 2024-03-25T18:25:09.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.911+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2950141d50 con 0x7f29500b1730 2024-03-25T18:25:09.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.911+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2940001b00 con 0x7f29500b1730 2024-03-25T18:25:09.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.911+0000 7f293effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.911+0000 7f293effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:09.912 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.911+0000 7f293effd640 10 monclient(hunting): dump: 2024-03-25T18:25:09.912 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:09.912 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:09.913 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:09.913 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:09.913 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.913 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:09.913 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:09.913 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:09.913 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:09.913 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:09.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 459988471 0 0) 0x7f2950141d50 con 0x7f29500b1730 2024-03-25T18:25:09.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 >> v1:172.21.15.162:6790/0 conn(0x7f295013b8b0 legacy=0x7f2950142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 >> v1:172.21.15.162:6789/0 conn(0x7f295014be90 legacy=0x7f29501d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:09.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 1 monclient: found mon.b 2024-03-25T18:25:09.918 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29501d8600 con 0x7f29500b1730 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f293effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 10 monclient: _renew_subs 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.191:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2950143330 con 0x7f29500b1730 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 10 monclient: _renew_subs 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.191:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29501425c0 con 0x7f29500b1730 2024-03-25T18:25:09.919 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.916+0000 7f2954e55640 1 librados: init done 2024-03-25T18:25:09.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.918+0000 7f2954e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.918+0000 7f2954e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.918+0000 7f2954e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:25:09.920 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:09.918+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29501d7d40 con 0x7f29500b1730 2024-03-25T18:25:10.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.206+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2940004a90 con 0x7f29500b1730 2024-03-25T18:25:10.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.206+0000 7f293effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:10.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.206+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f2940004da0 con 0x7f29500b1730 2024-03-25T18:25:10.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.206+0000 7f293effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:10.207 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.206+0000 7f293effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.206+0000 7f293effd640 10 monclient: dump: 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:10.208 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.206+0000 7f294cd99640 4 set_mon_vals no callback set 2024-03-25T18:25:10.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.207+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f294001ad70 con 0x7f29500b1730 2024-03-25T18:25:10.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.207+0000 7f294dd9b640 10 monclient: get_auth_request con 0x7f293006dfd0 auth_method 0 2024-03-25T18:25:10.209 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.208+0000 7f293effd640 10 client.4717.objecter ms_handle_connect 0x7f293006dfd0 2024-03-25T18:25:10.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.212+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7f29400547f0 con 0x7f29500b1730 2024-03-25T18:25:10.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.212+0000 7f293effd640 10 client.4717.objecter ms_dispatch 0x7f2950000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:10.213 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.212+0000 7f293effd640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:10.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.212+0000 7f293effd640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:10.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.212+0000 7f293effd640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T18:25:10.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.213+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2940050300 con 0x7f29500b1730 2024-03-25T18:25:10.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.213+0000 7f293effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:10.214 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.213+0000 7f293effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:10.463 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.462+0000 7f2954e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:10.464 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.462+0000 7f2954e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:10.464 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.462+0000 7f2954e55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.191:6789/0 2024-03-25T18:25:10.464 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.462+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 --> v1:172.21.15.191:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f295013c2c0 con 0x7f29500b1730 2024-03-25T18:25:10.496 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.495+0000 7f293effd640 1 -- 172.21.15.162:0/2483584831 <== mon.1 v1:172.21.15.191:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7f2940052230 con 0x7f29500b1730 2024-03-25T18:25:10.496 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.495+0000 7f293effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:10.497 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.495+0000 7f293effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:10.497 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:25:10.497 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:10.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.497+0000 7f2954e55640 10 librados: watch_flush enter 2024-03-25T18:25:10.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.497+0000 7f2954e55640 10 librados: watch_flush exit 2024-03-25T18:25:10.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.497+0000 7f2954e55640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T18:25:10.499 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 10 client.4717.objecter successfully canceled tick 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 >> v1:172.21.15.191:6800/1552703795 conn(0x7f293006dfd0 legacy=0x7f2930070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 10 monclient: shutdown 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 >> v1:172.21.15.191:6789/0 conn(0x7f29500b1730 legacy=0x7f29501405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 shutdown_connections 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 >> 172.21.15.162:0/2483584831 conn(0x7f29500617a0 msgr2=0x7f29500b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 shutdown_connections 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 1 -- 172.21.15.162:0/2483584831 wait complete. 2024-03-25T18:25:10.500 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:10.498+0000 7f2954e55640 1 librados: shutdown 2024-03-25T18:25:10.510 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:17.318 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:17.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.433+0000 7f6c43193640 10 monclient: get_monmap_and_config 2024-03-25T18:25:17.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.433+0000 7f6c43193640 10 monclient: build_initial_monmap 2024-03-25T18:25:17.434 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.433+0000 7f6c43193640 10 monclient: monmap: 2024-03-25T18:25:17.434 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:17.434 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:17.434 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:25:17.434158+0000 2024-03-25T18:25:17.434 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:25:17.434158+0000 2024-03-25T18:25:17.434 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:17.435 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:17.435 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:17.435 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:25:17.435 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:25:17.435 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.433+0000 7f6c43193640 1 Processor -- start 2024-03-25T18:25:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 1 -- start start 2024-03-25T18:25:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 10 monclient: init 2024-03-25T18:25:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 10 monclient(hunting): picked mon.noname-b con 0x7f6c3c0b4d60 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:17.436 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3c0b1730 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:17.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3c14be90 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:17.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0af940 con 0x7f6c3c14be90 2024-03-25T18:25:17.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0b0a70 con 0x7f6c3c0b1730 2024-03-25T18:25:17.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c0b0d50 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:17.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.434+0000 7f6c43193640 10 monclient(hunting): authenticate will time out at 8216.708302s 2024-03-25T18:25:17.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.435+0000 7f6c40f2f640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f6c3c0b4d60 0x7f6c3c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:38022/0 (socket says 172.21.15.162:38022) 2024-03-25T18:25:17.437 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.435+0000 7f6c40f2f640 1 -- 172.21.15.162:0/440579315 learned_addr learned my addr 172.21.15.162:0/440579315 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.454+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478131357 0 0) 0x7f6c3c0af940 con 0x7f6c3c14be90 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.454+0000 7f6c3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.454+0000 7f6c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.454+0000 7f6c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.454+0000 7f6c3b7fe640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.454+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c2c003520 con 0x7f6c3c14be90 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.454+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317929770 0 0) 0x7f6c3c0b0a70 con 0x7f6c3c0b1730 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.455+0000 7f6c3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.455+0000 7f6c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.455+0000 7f6c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.455+0000 7f6c3b7fe640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:25:17.456 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.455+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c0af940 con 0x7f6c3c0b1730 2024-03-25T18:25:17.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.457+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459973045 0 0) 0x7f6c3c0b0d50 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.457+0000 7f6c3b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.457+0000 7f6c3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.457+0000 7f6c3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.457+0000 7f6c3b7fe640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:25:17.459 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.457+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c0b0a70 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.461 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2060809346 0 0) 0x7f6c2c003520 con 0x7f6c3c14be90 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 >> v1:172.21.15.162:6790/0 conn(0x7f6c3c0b4d60 legacy=0x7f6c3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 >> v1:172.21.15.191:6789/0 conn(0x7f6c3c0b1730 legacy=0x7f6c3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c0b1030 con 0x7f6c3c14be90 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6c34001a10 con 0x7f6c3c14be90 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c3b7fe640 10 monclient: dump: 2024-03-25T18:25:17.462 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c43193640 5 monclient: authenticate success, global_id 4740 2024-03-25T18:25:17.463 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.460+0000 7f6c43193640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:17.467 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.466+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c34001b70 con 0x7f6c3c14be90 2024-03-25T18:25:17.467 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.466+0000 7f6c3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:17.467 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.466+0000 7f6c3b7fe640 1 -- 172.21.15.162:0/440579315 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6c34001f80 con 0x7f6c3c14be90 2024-03-25T18:25:17.467 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.466+0000 7f6c43193640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:17.467 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.466+0000 7f6c43193640 4 set_mon_vals no callback set 2024-03-25T18:25:17.467 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.466+0000 7f6c3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.466+0000 7f6c3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.466+0000 7f6c3b7fe640 10 monclient: dump: 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:17.468 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.467+0000 7f6c43193640 10 monclient: shutdown 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.467+0000 7f6c43193640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.467+0000 7f6c43193640 1 -- 172.21.15.162:0/440579315 >> v1:172.21.15.162:6789/0 conn(0x7f6c3c14be90 legacy=0x7f6c3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.467+0000 7f6c43193640 1 -- 172.21.15.162:0/440579315 shutdown_connections 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.467+0000 7f6c43193640 1 -- 172.21.15.162:0/440579315 >> 172.21.15.162:0/440579315 conn(0x7f6c3c0617a0 msgr2=0x7f6c3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.467+0000 7f6c43193640 1 -- 172.21.15.162:0/440579315 shutdown_connections 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.467+0000 7f6c43193640 1 -- 172.21.15.162:0/440579315 wait complete. 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.468+0000 7f6c43193640 10 monclient: build_initial_monmap 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.468+0000 7f6c43193640 10 monclient: monmap: 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:17.470 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.468+0000 7f6c43193640 1 librados: starting msgr at 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.468+0000 7f6c43193640 1 librados: starting objecter 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.469+0000 7f6c43193640 1 Processor -- start 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.469+0000 7f6c43193640 1 -- start start 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.469+0000 7f6c43193640 1 librados: setting wanted keys 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.469+0000 7f6c43193640 1 librados: calling monclient init 2024-03-25T18:25:17.471 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.469+0000 7f6c43193640 10 monclient: init 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3c0b1730 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 10 monclient(hunting): picked mon.noname-b con 0x7f6c3c0b4d60 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3c14be90 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c1d87c0 con 0x7f6c3c0b1730 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c138820 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c1389b0 con 0x7f6c3c14be90 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:17.473 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.470+0000 7f6c43193640 10 monclient(hunting): authenticate will time out at 8216.743941s 2024-03-25T18:25:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.482+0000 7f6c41730640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7f6c3c14be90 0x7f6c3c1e22a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:38048/0 (socket says 172.21.15.162:38048) 2024-03-25T18:25:17.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.482+0000 7f6c41730640 1 -- 172.21.15.162:0/4142414793 learned_addr learned my addr 172.21.15.162:0/4142414793 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:17.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.483+0000 7f6c397fa640 10 client.?.objecter ms_handle_connect 0x7f6c3c14be90 2024-03-25T18:25:17.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.483+0000 7f6c397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:17.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.488+0000 7f6c397fa640 10 client.?.objecter ms_handle_connect 0x7f6c3c0b4d60 2024-03-25T18:25:17.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.488+0000 7f6c397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:17.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.493+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764112033 0 0) 0x7f6c3c1389b0 con 0x7f6c3c14be90 2024-03-25T18:25:17.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.493+0000 7f6c397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.493+0000 7f6c397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.493+0000 7f6c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.493+0000 7f6c397fa640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:25:17.494 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.493+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c20004eb0 con 0x7f6c3c14be90 2024-03-25T18:25:17.503 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.502+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059278012 0 0) 0x7f6c3c138820 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.502+0000 7f6c397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.502+0000 7f6c397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.502+0000 7f6c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.502+0000 7f6c397fa640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:25:17.504 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.502+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c3c1389b0 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.505+0000 7f6c397fa640 10 client.?.objecter ms_handle_connect 0x7f6c3c0b1730 2024-03-25T18:25:17.506 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.505+0000 7f6c397fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.512+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2599501454 0 0) 0x7f6c20004eb0 con 0x7f6c3c14be90 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.512+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c3c138820 con 0x7f6c3c14be90 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.512+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6c34001ec0 con 0x7f6c3c14be90 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.512+0000 7f6c397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.512+0000 7f6c397fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.512+0000 7f6c397fa640 10 monclient(hunting): dump: 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:17.513 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:17.514 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:17.514 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:17.514 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:17.514 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:17.514 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.512+0000 7f6c397fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:17.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.524+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097774280 0 0) 0x7f6c3c1d87c0 con 0x7f6c3c0b1730 2024-03-25T18:25:17.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.524+0000 7f6c397fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:17.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.524+0000 7f6c397fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:17.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.524+0000 7f6c397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:17.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.524+0000 7f6c397fa640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:25:17.525 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.524+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c20004eb0 con 0x7f6c3c0b1730 2024-03-25T18:25:17.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.525+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 913306577 0 0) 0x7f6c3c1389b0 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.525+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c3c1d87c0 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.525+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6c28001970 con 0x7f6c3c0b4d60 2024-03-25T18:25:17.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.525+0000 7f6c397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:17.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.525+0000 7f6c397fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:17.526 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.525+0000 7f6c397fa640 10 monclient(hunting): dump: 2024-03-25T18:25:17.526 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:17.527 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:17.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4222837656 0 0) 0x7f6c3c138820 con 0x7f6c3c14be90 2024-03-25T18:25:17.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 >> v1:172.21.15.191:6789/0 conn(0x7f6c3c0b4d60 legacy=0x7f6c3c1d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 >> v1:172.21.15.162:6789/0 conn(0x7f6c3c0b1730 legacy=0x7f6c3c138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.533 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 1 monclient: found mon.c 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c1405f0 con 0x7f6c3c14be90 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 10 monclient: _renew_subs 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c1407e0 con 0x7f6c3c14be90 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 10 client.4727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 10 monclient: _renew_subs 2024-03-25T18:25:17.534 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:17.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c1e3ba0 con 0x7f6c3c14be90 2024-03-25T18:25:17.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.532+0000 7f6c43193640 1 librados: init done 2024-03-25T18:25:17.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.533+0000 7f6c43193640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:17.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.533+0000 7f6c43193640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:17.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.533+0000 7f6c43193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:17.535 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.533+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c1d7f00 con 0x7f6c3c14be90 2024-03-25T18:25:17.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.831+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c34003c20 con 0x7f6c3c14be90 2024-03-25T18:25:17.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.831+0000 7f6c397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:17.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.831+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7f6c34002100 con 0x7f6c3c14be90 2024-03-25T18:25:17.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.831+0000 7f6c397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:17.832 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.831+0000 7f6c397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.831+0000 7f6c397fa640 10 monclient: dump: 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:17.833 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.831+0000 7f6c3affd640 4 set_mon_vals no callback set 2024-03-25T18:25:17.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.832+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7f6c340040f0 con 0x7f6c3c14be90 2024-03-25T18:25:17.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.833+0000 7f6c40f2f640 10 monclient: get_auth_request con 0x7f6c2006dfb0 auth_method 0 2024-03-25T18:25:17.834 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.833+0000 7f6c397fa640 10 client.4727.objecter ms_handle_connect 0x7f6c2006dfb0 2024-03-25T18:25:17.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.838+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7f6c34054a40 con 0x7f6c3c14be90 2024-03-25T18:25:17.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.838+0000 7f6c397fa640 10 client.4727.objecter ms_dispatch 0x7f6c3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:17.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.838+0000 7f6c397fa640 3 client.4727.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:17.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.838+0000 7f6c397fa640 3 client.4727.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:17.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.838+0000 7f6c397fa640 20 client.4727.objecter dump_active .. 0 homeless 2024-03-25T18:25:17.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.838+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c34050550 con 0x7f6c3c14be90 2024-03-25T18:25:17.839 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.838+0000 7f6c397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:17.840 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:17.838+0000 7f6c397fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:18.089 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.088+0000 7f6c43193640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:18.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.088+0000 7f6c43193640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:18.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.088+0000 7f6c43193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:18.090 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.088+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c3c0b5ac0 con 0x7f6c3c14be90 2024-03-25T18:25:18.118 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.117+0000 7f6c397fa640 1 -- 172.21.15.162:0/4142414793 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7f6c34052480 con 0x7f6c3c14be90 2024-03-25T18:25:18.118 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.117+0000 7f6c397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:18.118 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.117+0000 7f6c397fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:18.118 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:25:18.118 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:18.120 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 10 librados: watch_flush enter 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 10 librados: watch_flush exit 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 20 client.4727.objecter shutdown clearing up homeless session... 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 10 client.4727.objecter successfully canceled tick 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 >> v1:172.21.15.191:6800/1552703795 conn(0x7f6c2006dfb0 legacy=0x7f6c20070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 10 monclient: shutdown 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 >> v1:172.21.15.162:6790/0 conn(0x7f6c3c14be90 legacy=0x7f6c3c1e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 shutdown_connections 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 >> 172.21.15.162:0/4142414793 conn(0x7f6c3c0617a0 msgr2=0x7f6c3c0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 shutdown_connections 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 1 -- 172.21.15.162:0/4142414793 wait complete. 2024-03-25T18:25:18.121 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:18.119+0000 7f6c43193640 1 librados: shutdown 2024-03-25T18:25:18.132 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:24.180 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:36.407 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.185 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:36.412 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.216 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:36.444 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.231 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:36.447 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.238 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:36.465 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.245 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:36.461 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.246 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:36.473 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.275 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:36.491 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.297 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:36.525 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:24.939 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.054+0000 7fe57d874640 10 monclient: get_monmap_and_config 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.054+0000 7fe57d874640 10 monclient: build_initial_monmap 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.054+0000 7fe57d874640 10 monclient: monmap: 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:25:25.055891+0000 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:25:25.055891+0000 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:25:25.056 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:25.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.055+0000 7fe57d874640 1 Processor -- start 2024-03-25T18:25:25.057 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.055+0000 7fe57d874640 1 -- start start 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.055+0000 7fe57d874640 10 monclient: init 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 10 monclient(hunting): picked mon.noname-a con 0x7fe5780a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 10 monclient(hunting): picked mon.noname-c con 0x7fe57813f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 10 monclient(hunting): picked mon.noname-b con 0x7fe5781430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe578142520 con 0x7fe5780a98f0 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780b28a0 con 0x7fe57813f710 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780b2a30 con 0x7fe5781430c0 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:25.058 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.056+0000 7fe57d874640 10 monclient(hunting): authenticate will time out at 8224.330152s 2024-03-25T18:25:25.063 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.062+0000 7fe576ffd640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7fe5780a98f0 0x7fe5780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:42194/0 (socket says 172.21.15.162:42194) 2024-03-25T18:25:25.063 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.062+0000 7fe576ffd640 1 -- 172.21.15.162:0/3020037723 learned_addr learned my addr 172.21.15.162:0/3020037723 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:25.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.072+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413947023 0 0) 0x7fe578142520 con 0x7fe5780a98f0 2024-03-25T18:25:25.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.072+0000 7fe5767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:25.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.072+0000 7fe5767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:25.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.072+0000 7fe5767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:25.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.072+0000 7fe5767fc640 10 monclient(hunting): my global_id is 4746 2024-03-25T18:25:25.073 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.072+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe564003520 con 0x7fe5780a98f0 2024-03-25T18:25:25.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.083+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178452820 0 0) 0x7fe5780b2a30 con 0x7fe5781430c0 2024-03-25T18:25:25.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.083+0000 7fe5767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:25.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.083+0000 7fe5767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:25.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.083+0000 7fe5767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:25.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.083+0000 7fe5767fc640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:25:25.085 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.083+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe578142520 con 0x7fe5781430c0 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224069255 0 0) 0x7fe564003520 con 0x7fe5780a98f0 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 >> v1:172.21.15.162:6790/0 conn(0x7fe5781430c0 legacy=0x7fe57814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 >> v1:172.21.15.191:6789/0 conn(0x7fe57813f710 legacy=0x7fe5781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 1 monclient: found mon.noname-a 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5780b44c0 con 0x7fe5780a98f0 2024-03-25T18:25:25.091 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe568001690 con 0x7fe5780a98f0 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe57d874640 5 monclient: authenticate success, global_id 4746 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe57d874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe5767fc640 10 monclient: dump: 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:25.092 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:25.093 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:25.093 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:25.093 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.089+0000 7fe57d874640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:25.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.106+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5680017f0 con 0x7fe5780a98f0 2024-03-25T18:25:25.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.106+0000 7fe5767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:25.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.106+0000 7fe5767fc640 1 -- 172.21.15.162:0/3020037723 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe568001c00 con 0x7fe5780a98f0 2024-03-25T18:25:25.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.106+0000 7fe57d874640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:25.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.106+0000 7fe57d874640 4 set_mon_vals no callback set 2024-03-25T18:25:25.107 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.106+0000 7fe5767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.106+0000 7fe5767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.106+0000 7fe5767fc640 10 monclient: dump: 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:25.108 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:25.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.107+0000 7fe57d874640 10 monclient: shutdown 2024-03-25T18:25:25.109 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.107+0000 7fe57d874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.107+0000 7fe57d874640 1 -- 172.21.15.162:0/3020037723 >> v1:172.21.15.162:6789/0 conn(0x7fe5780a98f0 legacy=0x7fe5780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.107+0000 7fe57d874640 1 -- 172.21.15.162:0/3020037723 shutdown_connections 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.107+0000 7fe57d874640 1 -- 172.21.15.162:0/3020037723 >> 172.21.15.162:0/3020037723 conn(0x7fe5780b7ea0 msgr2=0x7fe5780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.107+0000 7fe57d874640 1 -- 172.21.15.162:0/3020037723 shutdown_connections 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.107+0000 7fe57d874640 1 -- 172.21.15.162:0/3020037723 wait complete. 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 10 monclient: build_initial_monmap 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 10 monclient: monmap: 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:25:25.110 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 1 librados: starting msgr at 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 1 librados: starting objecter 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 1 Processor -- start 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 1 -- start start 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 1 librados: setting wanted keys 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 1 librados: calling monclient init 2024-03-25T18:25:25.111 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.108+0000 7fe57d874640 10 monclient: init 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 10 monclient(hunting): picked mon.noname-b con 0x7fe5780a98f0 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 10 monclient(hunting): picked mon.noname-a con 0x7fe57813f710 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 10 monclient(hunting): picked mon.noname-c con 0x7fe5781430c0 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780701b0 con 0x7fe57813f710 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe578070400 con 0x7fe5780a98f0 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5780706e0 con 0x7fe5781430c0 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:25.112 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.109+0000 7fe57d874640 10 monclient(hunting): authenticate will time out at 8224.383521s 2024-03-25T18:25:25.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.114+0000 7fe576ffd640 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe5780a98f0 0x7fe578070de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.162:60440/0 (socket says 172.21.15.162:60440) 2024-03-25T18:25:25.115 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.114+0000 7fe576ffd640 1 -- 172.21.15.162:0/943458845 learned_addr learned my addr 172.21.15.162:0/943458845 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:25.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.116+0000 7fe574ff9640 10 client.?.objecter ms_handle_connect 0x7fe5780a98f0 2024-03-25T18:25:25.117 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.116+0000 7fe574ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:25.123 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.122+0000 7fe574ff9640 10 client.?.objecter ms_handle_connect 0x7fe57813f710 2024-03-25T18:25:25.124 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.122+0000 7fe574ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:25.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.125+0000 7fe574ff9640 10 client.?.objecter ms_handle_connect 0x7fe5781430c0 2024-03-25T18:25:25.126 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.125+0000 7fe574ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:25.138 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.137+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.1 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484501880 0 0) 0x7fe578070400 con 0x7fe5780a98f0 2024-03-25T18:25:25.138 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.137+0000 7fe574ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:25.138 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.137+0000 7fe574ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:25.138 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.137+0000 7fe574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:25.138 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.137+0000 7fe574ff9640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:25:25.138 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.137+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe558004eb0 con 0x7fe5780a98f0 2024-03-25T18:25:25.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.141+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689408588 0 0) 0x7fe5780706e0 con 0x7fe5781430c0 2024-03-25T18:25:25.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.141+0000 7fe574ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:25.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.141+0000 7fe574ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:25.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.141+0000 7fe574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:25.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.141+0000 7fe574ff9640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:25:25.143 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.141+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe578070400 con 0x7fe5781430c0 2024-03-25T18:25:25.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.151+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123957930 0 0) 0x7fe5780701b0 con 0x7fe57813f710 2024-03-25T18:25:25.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.151+0000 7fe574ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:25.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.151+0000 7fe574ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:25.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.151+0000 7fe574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:25.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.151+0000 7fe574ff9640 10 monclient(hunting): my global_id is 4749 2024-03-25T18:25:25.152 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.151+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5780706e0 con 0x7fe57813f710 2024-03-25T18:25:25.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.152+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.1 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1329845914 0 0) 0x7fe558004eb0 con 0x7fe5780a98f0 2024-03-25T18:25:25.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.152+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.191:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5780701b0 con 0x7fe5780a98f0 2024-03-25T18:25:25.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.152+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.1 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe568001c00 con 0x7fe5780a98f0 2024-03-25T18:25:25.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.152+0000 7fe574ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:25.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.152+0000 7fe574ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:25.153 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.152+0000 7fe574ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:25.153 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:25.153 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:25.154 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.152+0000 7fe574ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:25.163 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.162+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1234635297 0 0) 0x7fe578070400 con 0x7fe5781430c0 2024-03-25T18:25:25.163 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.162+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe558004eb0 con 0x7fe5781430c0 2024-03-25T18:25:25.163 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.162+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe570001b90 con 0x7fe5781430c0 2024-03-25T18:25:25.163 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.162+0000 7fe574ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:25.163 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.162+0000 7fe574ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:25.163 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.162+0000 7fe574ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:25.163 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:25.164 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:25.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.172+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 593904195 0 0) 0x7fe5780706e0 con 0x7fe57813f710 2024-03-25T18:25:25.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.172+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe578070400 con 0x7fe57813f710 2024-03-25T18:25:25.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.172+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe560001860 con 0x7fe57813f710 2024-03-25T18:25:25.173 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.172+0000 7fe574ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.172+0000 7fe574ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.172+0000 7fe574ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:25.174 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:25.175 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:25.176 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 436008242 0 0) 0x7fe558004eb0 con 0x7fe5781430c0 2024-03-25T18:25:25.176 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 >> v1:172.21.15.191:6789/0 conn(0x7fe5780a98f0 legacy=0x7fe578070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.176 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 >> v1:172.21.15.162:6789/0 conn(0x7fe57813f710 legacy=0x7fe5781381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.176 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:25.176 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 1 monclient: found mon.c 2024-03-25T18:25:25.176 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:25.176 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5780709c0 con 0x7fe5781430c0 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe574ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 5 monclient: authenticate success, global_id 4733 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 10 monclient: _renew_subs 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57806f210 con 0x7fe5781430c0 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 10 monclient: _renew_subs 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57806f730 con 0x7fe5781430c0 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.173+0000 7fe57d874640 1 librados: init done 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.175+0000 7fe57d874640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:25.177 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.175+0000 7fe57d874640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:25.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.175+0000 7fe57d874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:25.178 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.175+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe578074cc0 con 0x7fe5781430c0 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.481+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe570002040 con 0x7fe5781430c0 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.481+0000 7fe574ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.481+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7fe570003b10 con 0x7fe5781430c0 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.481+0000 7fe574ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.481+0000 7fe574ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.481+0000 7fe574ff9640 10 monclient: dump: 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.483 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:25.484 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:25.484 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:25.484 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:25.484 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:25.484 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:25.484 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:25.484 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.481+0000 7fe575ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:25.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.482+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7fe570019ae0 con 0x7fe5781430c0 2024-03-25T18:25:25.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.483+0000 7fe576ffd640 10 monclient: get_auth_request con 0x7fe55806e110 auth_method 0 2024-03-25T18:25:25.485 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.483+0000 7fe574ff9640 10 client.4733.objecter ms_handle_connect 0x7fe55806e110 2024-03-25T18:25:25.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.488+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7fe5700537d0 con 0x7fe5781430c0 2024-03-25T18:25:25.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.488+0000 7fe574ff9640 10 client.4733.objecter ms_dispatch 0x7fe578000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:25.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.488+0000 7fe574ff9640 3 client.4733.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:25.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.488+0000 7fe574ff9640 3 client.4733.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:25.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.488+0000 7fe574ff9640 20 client.4733.objecter dump_active .. 0 homeless 2024-03-25T18:25:25.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.488+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe57004f110 con 0x7fe5781430c0 2024-03-25T18:25:25.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.488+0000 7fe574ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:25.489 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.488+0000 7fe574ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:25.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.703+0000 7fe57d874640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:25.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.703+0000 7fe57d874640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:25.705 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.703+0000 7fe57d874640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.162:6790/0 2024-03-25T18:25:25.705 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.703+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 --> v1:172.21.15.162:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5780627b0 con 0x7fe5781430c0 2024-03-25T18:25:25.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.732+0000 7fe574ff9640 1 -- 172.21.15.162:0/943458845 <== mon.2 v1:172.21.15.162:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7fe570051210 con 0x7fe5781430c0 2024-03-25T18:25:25.733 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.732+0000 7fe574ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:25.734 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.732+0000 7fe574ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:25.734 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:25:25.734 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:25.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.734+0000 7fe57d874640 10 librados: watch_flush enter 2024-03-25T18:25:25.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.734+0000 7fe57d874640 10 librados: watch_flush exit 2024-03-25T18:25:25.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.734+0000 7fe57d874640 20 client.4733.objecter shutdown clearing up homeless session... 2024-03-25T18:25:25.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.734+0000 7fe57d874640 10 client.4733.objecter successfully canceled tick 2024-03-25T18:25:25.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.734+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 >> v1:172.21.15.191:6800/1552703795 conn(0x7fe55806e110 legacy=0x7fe5580705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.734+0000 7fe57d874640 10 monclient: shutdown 2024-03-25T18:25:25.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.734+0000 7fe57d874640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:25.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.734+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 >> v1:172.21.15.162:6790/0 conn(0x7fe5781430c0 legacy=0x7fe5781e77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:25.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.735+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 shutdown_connections 2024-03-25T18:25:25.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.735+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 >> 172.21.15.162:0/943458845 conn(0x7fe5780b7ea0 msgr2=0x7fe578074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:25.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.735+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 shutdown_connections 2024-03-25T18:25:25.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.735+0000 7fe57d874640 1 -- 172.21.15.162:0/943458845 wait complete. 2024-03-25T18:25:25.737 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:25.735+0000 7fe57d874640 1 librados: shutdown 2024-03-25T18:25:25.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:32.555 DEBUG:teuthology.orchestra.run.smithi162:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:32.675 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.674+0000 7ff7dad3a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:32.675 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.674+0000 7ff7dad3a640 10 monclient: build_initial_monmap 2024-03-25T18:25:32.675 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.674+0000 7ff7dad3a640 10 monclient: monmap: 2024-03-25T18:25:32.675 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:32.675 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:32.675 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:25:32.675249+0000 2024-03-25T18:25:32.676 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:25:32.675249+0000 2024-03-25T18:25:32.676 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:32.676 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:32.676 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:32.676 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-c 2024-03-25T18:25:32.676 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-b 2024-03-25T18:25:32.676 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:32.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.675+0000 7ff7dad3a640 1 Processor -- start 2024-03-25T18:25:32.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.675+0000 7ff7dad3a640 1 -- start start 2024-03-25T18:25:32.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.675+0000 7ff7dad3a640 10 monclient: init 2024-03-25T18:25:32.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.675+0000 7ff7dad3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:32.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.675+0000 7ff7dad3a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:32.677 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.675+0000 7ff7dad3a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7d40a98f0 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:32.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.675+0000 7ff7dad3a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7d413f710 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:32.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.676+0000 7ff7dad3a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7d41430c0 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:32.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.676+0000 7ff7dad3a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d4142520 con 0x7ff7d41430c0 2024-03-25T18:25:32.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.676+0000 7ff7dad3a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d40b28a0 con 0x7ff7d413f710 2024-03-25T18:25:32.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.676+0000 7ff7dad3a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d40b2a30 con 0x7ff7d40a98f0 2024-03-25T18:25:32.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.676+0000 7ff7dad3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:32.678 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.676+0000 7ff7dad3a640 10 monclient(hunting): authenticate will time out at 8231.949789s 2024-03-25T18:25:32.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.686+0000 7ff7d92d7640 1 --1- >> v1:172.21.15.162:6789/0 conn(0x7ff7d41430c0 0x7ff7d414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.162:6789/0 says I am v1:172.21.15.162:46640/0 (socket says 172.21.15.162:46640) 2024-03-25T18:25:32.687 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.686+0000 7ff7d92d7640 1 -- 172.21.15.162:0/1422474011 learned_addr learned my addr 172.21.15.162:0/1422474011 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:32.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.703+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539521069 0 0) 0x7ff7d4142520 con 0x7ff7d41430c0 2024-03-25T18:25:32.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.703+0000 7ff7d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.703+0000 7ff7d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.703+0000 7ff7d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.703+0000 7ff7d37fe640 10 monclient(hunting): my global_id is 4752 2024-03-25T18:25:32.704 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.703+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c4003520 con 0x7ff7d41430c0 2024-03-25T18:25:32.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.709+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 <== mon.2 v1:172.21.15.162:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964232341 0 0) 0x7ff7d40b2a30 con 0x7ff7d40a98f0 2024-03-25T18:25:32.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.709+0000 7ff7d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.710 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.709+0000 7ff7d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.709+0000 7ff7d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.709+0000 7ff7d37fe640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:25:32.711 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.709+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 --> v1:172.21.15.162:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d4142520 con 0x7ff7d40a98f0 2024-03-25T18:25:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.710+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4186269359 0 0) 0x7ff7c4003520 con 0x7ff7d41430c0 2024-03-25T18:25:32.712 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.710+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 >> v1:172.21.15.162:6790/0 conn(0x7ff7d40a98f0 legacy=0x7ff7d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.710+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 >> v1:172.21.15.191:6789/0 conn(0x7ff7d413f710 legacy=0x7ff7d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.710+0000 7ff7d37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.710+0000 7ff7d37fe640 1 monclient: found mon.noname-a 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.710+0000 7ff7d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.162:6789/0 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d40b44c0 con 0x7ff7d41430c0 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff7cc001a10 con 0x7ff7d41430c0 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7dad3a640 5 monclient: authenticate success, global_id 4752 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7dad3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:32.713 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7d37fe640 10 monclient: dump: 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:32.714 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.711+0000 7ff7dad3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:32.735 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.734+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 <== mon.0 v1:172.21.15.162:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7cc001b70 con 0x7ff7d41430c0 2024-03-25T18:25:32.735 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.734+0000 7ff7d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:32.735 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.734+0000 7ff7d37fe640 1 -- 172.21.15.162:0/1422474011 <== mon.0 v1:172.21.15.162:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff7cc001f80 con 0x7ff7d41430c0 2024-03-25T18:25:32.735 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.734+0000 7ff7d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.735 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.734+0000 7ff7d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.734+0000 7ff7d37fe640 10 monclient: dump: 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.734+0000 7ff7dad3a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:32.736 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.734+0000 7ff7dad3a640 4 set_mon_vals no callback set 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.735+0000 7ff7dad3a640 10 monclient: shutdown 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.735+0000 7ff7dad3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.735+0000 7ff7dad3a640 1 -- 172.21.15.162:0/1422474011 >> v1:172.21.15.162:6789/0 conn(0x7ff7d41430c0 legacy=0x7ff7d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.735+0000 7ff7dad3a640 1 -- 172.21.15.162:0/1422474011 shutdown_connections 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.735+0000 7ff7dad3a640 1 -- 172.21.15.162:0/1422474011 >> 172.21.15.162:0/1422474011 conn(0x7ff7d40b7ea0 msgr2=0x7ff7d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.735+0000 7ff7dad3a640 1 -- 172.21.15.162:0/1422474011 shutdown_connections 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.735+0000 7ff7dad3a640 1 -- 172.21.15.162:0/1422474011 wait complete. 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 10 monclient: build_initial_monmap 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 10 monclient: monmap: 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:epoch 0 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:last_changed 0.000000 2024-03-25T18:25:32.738 INFO:teuthology.misc.health.smithi162.stderr:created 0.000000 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 1 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.noname-a 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.noname-b 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.noname-c 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 1 librados: starting msgr at 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 1 librados: starting objecter 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 1 Processor -- start 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 1 -- start start 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 1 librados: setting wanted keys 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 1 librados: calling monclient init 2024-03-25T18:25:32.739 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.736+0000 7ff7dad3a640 10 monclient: init 2024-03-25T18:25:32.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.737+0000 7ff7dad3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:32.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.737+0000 7ff7dad3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:32.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.737+0000 7ff7dad3a640 10 monclient(hunting): picked mon.noname-a con 0x7ff7d40a98f0 addr v1:172.21.15.162:6789/0 2024-03-25T18:25:32.740 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.737+0000 7ff7dad3a640 10 monclient(hunting): picked mon.noname-c con 0x7ff7d413f710 addr v1:172.21.15.162:6790/0 2024-03-25T18:25:32.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.737+0000 7ff7dad3a640 10 monclient(hunting): picked mon.noname-b con 0x7ff7d41430c0 addr v1:172.21.15.191:6789/0 2024-03-25T18:25:32.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.737+0000 7ff7dad3a640 1 -- --> v1:172.21.15.162:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d413b410 con 0x7ff7d40a98f0 2024-03-25T18:25:32.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.738+0000 7ff7dad3a640 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d413b010 con 0x7ff7d41430c0 2024-03-25T18:25:32.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.738+0000 7ff7dad3a640 1 -- --> v1:172.21.15.162:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d4135230 con 0x7ff7d413f710 2024-03-25T18:25:32.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.738+0000 7ff7dad3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:32.741 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.738+0000 7ff7dad3a640 10 monclient(hunting): authenticate will time out at 8232.011750s 2024-03-25T18:25:32.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.743+0000 7ff7d3fff640 1 --1- >> v1:172.21.15.162:6790/0 conn(0x7ff7d413f710 0x7ff7d41dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.162:6790/0 says I am v1:172.21.15.162:54422/0 (socket says 172.21.15.162:54422) 2024-03-25T18:25:32.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.743+0000 7ff7d3fff640 1 -- 172.21.15.162:0/276811229 learned_addr learned my addr 172.21.15.162:0/276811229 (peer_addr_for_me v1:172.21.15.162:0/0) 2024-03-25T18:25:32.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.743+0000 7ff7d17fa640 10 client.?.objecter ms_handle_connect 0x7ff7d413f710 2024-03-25T18:25:32.744 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.743+0000 7ff7d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:32.747 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.746+0000 7ff7d17fa640 10 client.?.objecter ms_handle_connect 0x7ff7d41430c0 2024-03-25T18:25:32.747 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.746+0000 7ff7d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:32.751 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.750+0000 7ff7d17fa640 10 client.?.objecter ms_handle_connect 0x7ff7d40a98f0 2024-03-25T18:25:32.751 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.750+0000 7ff7d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:32.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.762+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220923919 0 0) 0x7ff7d413b410 con 0x7ff7d40a98f0 2024-03-25T18:25:32.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.762+0000 7ff7d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:32.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.762+0000 7ff7d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:32.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.762+0000 7ff7d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:32.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.762+0000 7ff7d17fa640 10 monclient(hunting): my global_id is 4755 2024-03-25T18:25:32.763 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.762+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 --> v1:172.21.15.162:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b8004eb0 con 0x7ff7d40a98f0 2024-03-25T18:25:32.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.769+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1233164058 0 0) 0x7ff7b8004eb0 con 0x7ff7d40a98f0 2024-03-25T18:25:32.770 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.769+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 --> v1:172.21.15.162:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7d413b410 con 0x7ff7d40a98f0 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.769+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff7c8001930 con 0x7ff7d40a98f0 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.769+0000 7ff7d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.769+0000 7ff7d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.769+0000 7ff7d17fa640 10 monclient(hunting): dump: 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:32.771 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.769+0000 7ff7d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:32.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1757316821 0 0) 0x7ff7d413b410 con 0x7ff7d40a98f0 2024-03-25T18:25:32.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 >> v1:172.21.15.162:6790/0 conn(0x7ff7d413f710 legacy=0x7ff7d41dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 >> v1:172.21.15.191:6789/0 conn(0x7ff7d41430c0 legacy=0x7ff7d41db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.780 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 1 monclient: found mon.a 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 --> v1:172.21.15.162:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d41353c0 con 0x7ff7d40a98f0 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 5 monclient: authenticate success, global_id 4755 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 10 monclient: _renew_subs 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 --> v1:172.21.15.162:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d40b78e0 con 0x7ff7d40a98f0 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 10 client.4755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 10 monclient: _renew_subs 2024-03-25T18:25:32.781 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:25:32.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 --> v1:172.21.15.162:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d41315f0 con 0x7ff7d40a98f0 2024-03-25T18:25:32.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.779+0000 7ff7dad3a640 1 librados: init done 2024-03-25T18:25:32.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.780+0000 7ff7dad3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:32.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.780+0000 7ff7dad3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:32.782 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.780+0000 7ff7dad3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:25:32.783 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:32.780+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d40627b0 con 0x7ff7d40a98f0 2024-03-25T18:25:33.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.038+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7c8003610 con 0x7ff7d40a98f0 2024-03-25T18:25:33.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.038+0000 7ff7d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:33.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.039+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1780083773 0 0) 0x7ff7c8003a60 con 0x7ff7d40a98f0 2024-03-25T18:25:33.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.039+0000 7ff7d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:33.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.039+0000 7ff7d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:33.040 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.039+0000 7ff7d17fa640 10 monclient: dump: 2024-03-25T18:25:33.040 INFO:teuthology.misc.health.smithi162.stderr:epoch 1 2024-03-25T18:25:33.040 INFO:teuthology.misc.health.smithi162.stderr:fsid e0e649a4-ed35-4995-8603-bf4aa2230a59 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr:last_changed 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr:created 2024-03-25T18:13:14.961507+0000 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr:election_strategy: 3 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr:0: v1:172.21.15.162:6789/0 mon.a 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr:1: v1:172.21.15.191:6789/0 mon.b 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr:2: v1:172.21.15.162:6790/0 mon.c 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr: 2024-03-25T18:25:33.041 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.039+0000 7ff7d2ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:33.042 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.039+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1153201299 0 0) 0x7ff7c8003c80 con 0x7ff7d40a98f0 2024-03-25T18:25:33.042 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.040+0000 7ff7d3fff640 10 monclient: get_auth_request con 0x7ff7b806dc70 auth_method 0 2024-03-25T18:25:33.042 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.040+0000 7ff7d17fa640 10 client.4755.objecter ms_handle_connect 0x7ff7b806dc70 2024-03-25T18:25:33.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.044+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3964583169 0 0) 0x7ff7c80535e0 con 0x7ff7d40a98f0 2024-03-25T18:25:33.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.044+0000 7ff7d17fa640 10 client.4755.objecter ms_dispatch 0x7ff7d4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:33.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.044+0000 7ff7d17fa640 3 client.4755.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:33.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.044+0000 7ff7d17fa640 3 client.4755.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:33.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.044+0000 7ff7d17fa640 20 client.4755.objecter dump_active .. 0 homeless 2024-03-25T18:25:33.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.044+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7c804f0f0 con 0x7ff7d40a98f0 2024-03-25T18:25:33.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.044+0000 7ff7d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:33.046 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.044+0000 7ff7d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:33.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.242+0000 7ff7dad3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:33.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.242+0000 7ff7dad3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:33.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.242+0000 7ff7dad3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.162:6789/0 2024-03-25T18:25:33.243 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.242+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 --> v1:172.21.15.162:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d40008d0 con 0x7ff7d40a98f0 2024-03-25T18:25:33.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.280+0000 7ff7d17fa640 1 -- 172.21.15.162:0/276811229 <== mon.0 v1:172.21.15.162:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1671471870) 0x7ff7c8051020 con 0x7ff7d40a98f0 2024-03-25T18:25:33.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.280+0000 7ff7d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:33.281 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.280+0000 7ff7d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:33.281 INFO:teuthology.misc.health.smithi162.stdout: 2024-03-25T18:25:33.281 INFO:teuthology.misc.health.smithi162.stdout:{"epoch":11,"fsid":"e0e649a4-ed35-4995-8603-bf4aa2230a59","created":"2024-03-25T18:14:04.675350+0000","modified":"2024-03-25T18:24:08.858730+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:08.858730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f567bd1-d9ae-41cb-b745-c7a607ecf767","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"450b2740-938f-45f6-96f9-7774f90ae10f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"24966f74-30d0-4d45-9c63-61d635d94d26","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"cf0e6d69-1049-4141-bfd3-3d590e7936f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9e7e1c42-7fbc-4ced-a70b-4556c30fbc66","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00c78472-8c62-45cc-bcd8-1ac19b38de5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"aad1d03e-a813-46d7-9e95-a932c7162531","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"96323cc0-8882-4860-b7f0-1e7fdc4e7d8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:33.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.282+0000 7ff7dad3a640 10 librados: watch_flush enter 2024-03-25T18:25:33.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 10 librados: watch_flush exit 2024-03-25T18:25:33.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 20 client.4755.objecter shutdown clearing up homeless session... 2024-03-25T18:25:33.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 10 client.4755.objecter successfully canceled tick 2024-03-25T18:25:33.284 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 >> v1:172.21.15.191:6800/1552703795 conn(0x7ff7b806dc70 legacy=0x7ff7b8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:33.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 10 monclient: shutdown 2024-03-25T18:25:33.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:33.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 >> v1:172.21.15.162:6789/0 conn(0x7ff7d40a98f0 legacy=0x7ff7d4130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:33.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 shutdown_connections 2024-03-25T18:25:33.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 >> 172.21.15.162:0/276811229 conn(0x7ff7d40b7ea0 msgr2=0x7ff7d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:33.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 shutdown_connections 2024-03-25T18:25:33.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 1 -- 172.21.15.162:0/276811229 wait complete. 2024-03-25T18:25:33.285 INFO:teuthology.misc.health.smithi162.stderr:2024-03-25T18:25:33.283+0000 7ff7dad3a640 1 librados: shutdown 2024-03-25T18:25:33.296 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:33.297 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:25:33.298 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:25:33.298 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:25:33.298 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:35.402 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:11:25.214 35334== 2024-03-25T18:25:35.402 INFO:tasks.ceph.osd.0.smithi162.stderr:==00:00:11:25.214 35334== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:35.436 INFO:tasks.ceph.osd.0.smithi162.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:39.400 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:25:39.401 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:25:39.401 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:25:39.402 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:41.260 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:11:31.055 35335== 2024-03-25T18:25:41.260 INFO:tasks.ceph.osd.1.smithi162.stderr:==00:00:11:31.055 35335== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:41.297 INFO:tasks.ceph.osd.1.smithi162.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:44.227 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:56.454 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.240 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:56.467 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.265 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:11:56.492 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.282 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:56.498 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.287 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:56.503 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.298 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:56.525 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.301 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:56.529 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.332 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:11:56.548 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.340 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:11:56.568 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:45.503 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:25:45.504 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:25:45.504 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:25:45.504 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:47.333 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:11:37.127 35336== 2024-03-25T18:25:47.333 INFO:tasks.ceph.osd.2.smithi162.stderr:==00:00:11:37.127 35336== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:47.369 INFO:tasks.ceph.osd.2.smithi162.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:51.605 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:25:51.606 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:25:51.606 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:25:51.606 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:53.338 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:11:43.127 35337== 2024-03-25T18:25:53.338 INFO:tasks.ceph.osd.3.smithi162.stderr:==00:00:11:43.127 35337== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:53.378 INFO:tasks.ceph.osd.3.smithi162.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:57.708 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:25:57.708 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:25:57.709 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:25:57.709 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:59.809 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:11:49.618 35462== 2024-03-25T18:25:59.809 INFO:tasks.ceph.osd.4.smithi191.stderr:==00:00:11:49.618 35462== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:59.845 INFO:tasks.ceph.osd.4.smithi191.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:03.810 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:26:03.811 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:26:03.811 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:26:03.811 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:04.250 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:12:16.477 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.254 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:12:16.482 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.274 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:12:16.501 35025== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.303 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:12:16.519 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.307 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:12:16.523 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.313 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:12:16.541 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.318 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:12:16.545 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.327 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:12:16.543 34919== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:04.337 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:12:16.565 35022== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:05.612 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:11:55.420 35463== 2024-03-25T18:26:05.612 INFO:tasks.ceph.osd.5.smithi191.stderr:==00:00:11:55.420 35463== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:05.651 INFO:tasks.ceph.osd.5.smithi191.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:09.913 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:26:09.913 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:26:09.913 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:26:09.914 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:11.672 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:12:01.478 35464== 2024-03-25T18:26:11.672 INFO:tasks.ceph.osd.6.smithi191.stderr:==00:00:12:01.478 35464== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:11.704 INFO:tasks.ceph.osd.6.smithi191.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:16.016 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:16.016 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:26:16.016 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:26:16.016 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:26:16.017 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:17.593 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:12:07.397 35465== 2024-03-25T18:26:17.594 INFO:tasks.ceph.osd.7.smithi191.stderr:==00:00:12:07.397 35465== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:17.624 INFO:tasks.ceph.osd.7.smithi191.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:22.118 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:26:22.119 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:26:22.119 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:26:22.119 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:26:22.119 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:22.147 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:26:22.147 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:26:22.148 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:22.149 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:26:22.147+0000 7f49c5854640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34917) UID: 0 2024-03-25T18:26:22.149 INFO:tasks.ceph.mgr.x.smithi191.stderr:2024-03-25T18:26:22.147+0000 7f49c5854640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:26:22.176 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:26:22.176 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:26:22.176 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:26:22.176 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:22.204 INFO:tasks.ceph.mon.a.smithi162.stderr:2024-03-25T18:26:22.191+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34988) UID: 0 2024-03-25T18:26:22.204 INFO:tasks.ceph.mon.a.smithi162.stderr:2024-03-25T18:26:22.193+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:28:13.952 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:14:26.180 35025== 2024-03-25T18:28:13.953 INFO:tasks.ceph.mon.a.smithi162.stderr:==00:00:14:26.180 35025== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:13.961 INFO:tasks.ceph.mon.a.smithi162.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:18.104 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:28:18.105 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:28:18.105 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:28:18.105 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:18.121 INFO:tasks.ceph.mon.c.smithi162.stderr:2024-03-25T18:28:18.109+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34990) UID: 0 2024-03-25T18:28:18.122 INFO:tasks.ceph.mon.c.smithi162.stderr:2024-03-25T18:28:18.110+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:28:18.696 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:14:30.924 35022== 2024-03-25T18:28:18.696 INFO:tasks.ceph.mon.c.smithi162.stderr:==00:00:14:30.924 35022== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:18.705 INFO:tasks.ceph.mon.c.smithi162.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:24.207 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:28:24.208 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:28:24.208 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:28:24.208 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:24.224 INFO:tasks.ceph.mon.b.smithi191.stderr:2024-03-25T18:28:24.212+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34915) UID: 0 2024-03-25T18:28:24.224 INFO:tasks.ceph.mon.b.smithi191.stderr:2024-03-25T18:28:24.213+0000 9f86640 -1 mon.b@1(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:28:24.850 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:14:37.065 34919== 2024-03-25T18:28:24.850 INFO:tasks.ceph.mon.b.smithi191.stderr:==00:00:14:37.065 34919== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:24.859 INFO:tasks.ceph.mon.b.smithi191.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:30.309 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:28:30.310 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:28:30.310 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:28:30.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:28:30.433 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:28:30.434 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:28:30.513 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:28:30.514 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:28:30.570 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:28:30.570 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:28:30.612 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:28:30.613 DEBUG:teuthology.orchestra.run.smithi162:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:28:30.713 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:28:30.713 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:28:30.826 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:28:30.826 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:28:30.874 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:28:30.875 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:28:30.924 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:28:30.924 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:28:30.972 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:28:30.974 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/data/mon.a.tgz 2024-03-25T18:28:30.975 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-25T18:28:30.990 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.76wMr4Y8kO 2024-03-25T18:28:30.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.76wMr4Y8kO 2024-03-25T18:28:31.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.76wMr4Y8kO 2024-03-25T18:28:31.175 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.76wMr4Y8kO is 133KB 2024-03-25T18:28:31.227 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.76wMr4Y8kO 2024-03-25T18:28:31.241 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/data/mon.c.tgz 2024-03-25T18:28:31.241 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-25T18:28:31.294 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.VnmF9S0YpE 2024-03-25T18:28:31.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.VnmF9S0YpE 2024-03-25T18:28:31.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.VnmF9S0YpE 2024-03-25T18:28:31.472 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.VnmF9S0YpE is 164KB 2024-03-25T18:28:31.523 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.VnmF9S0YpE 2024-03-25T18:28:31.537 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/data/mon.b.tgz 2024-03-25T18:28:31.538 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-03-25T18:28:31.555 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.GGcWq6DgCo 2024-03-25T18:28:31.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.GGcWq6DgCo 2024-03-25T18:28:31.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.GGcWq6DgCo 2024-03-25T18:28:31.748 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.GGcWq6DgCo is 155KB 2024-03-25T18:28:31.799 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.GGcWq6DgCo 2024-03-25T18:28:31.813 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:28:31.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:28:31.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:28:32.201 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:28:32.448 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:28:32.450 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/remote/smithi162/crash 2024-03-25T18:28:32.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:28:32.478 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/remote/smithi191/crash 2024-03-25T18:28:32.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:28:32.506 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:28:32.507 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:28:32.523 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:28:32.595 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:28:32.595 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35363.log 2024-03-25T18:28:32.595 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35072.log 2024-03-25T18:28:32.595 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.35363.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:28:32.595 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.35072.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:28:32.595 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35363.log.gz 2024-03-25T18:28:32.595 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- 86.8% /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35072.log.gz 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/valgrind/osd.7.log: /var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr: 73.6%/var/log/ceph/valgrind/mon.b.log: -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35232.log 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35112.log 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.35232.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.35112.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35232.log.gz 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.audit.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35112.log.gz 2024-03-25T18:28:32.596 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35192.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.4.log: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.35192.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35192.log.gz 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35152.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35316.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.35152.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35152.log.gz 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35272.log 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.35316.log: /var/log/ceph/ceph-client.admin.35272.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35316.log.gz 2024-03-25T18:28:32.597 INFO:teuthology.orchestra.run.smithi191.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35272.log.gz 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41556.log 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41634.log 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41556.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38898.log 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41556.log.gz 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39525.log 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40032.log 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.38898.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41634.log.gz 2024-03-25T18:28:32.601 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39915.log 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35360.log 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38898.log.gz 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39525.log.gz 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39915.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40032.log.gz 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40423.log 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.35360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39759.log 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39915.log.gz 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39564.log 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35360.log.gz 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39759.log: gzip 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40423.log.gz -5 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40149.log 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40306.log 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39759.log.gz 2024-03-25T18:28:32.602 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40149.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39564.log.gz 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40306.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40149.log.gz 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 86.3% 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40306.log.gz 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41244.log 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:28:32.603 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41907.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40737.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38859.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41244.log.gz 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39173.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41907.log.gz 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.38859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39486.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40737.log.gz 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39876.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38859.log.gz 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40542.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39173.log.gz 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39876.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39486.log.gz 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41049.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39720.log 2024-03-25T18:28:32.604 INFO:teuthology.orchestra.run.smithi162.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39876.log.gz 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41673.log 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40542.log.gz 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39720.log: gzip -5 --verbose 86.7% -- -- replaced with /var/log/ceph/ceph-client.admin.41049.log.gz /var/log/ceph/ceph-client.admin.41205.log 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40971.log 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41673.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39720.log.gz 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40071.log 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40971.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41673.log.gz 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39056.log 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41205.log.gz 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41283.log 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40971.log.gz 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39408.log 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40071.log.gz 2024-03-25T18:28:32.605 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38655.log 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39056.log.gz 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39408.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41283.log.gz 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.38655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40776.log 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39408.log.gz 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40659.log 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38655.log.gz 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40345.log 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40659.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39447.log 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40776.log.gz 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39837.log 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40345.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40659.log.gz 2024-03-25T18:28:32.606 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40227.log 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40345.log.gz 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39447.log.gz 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42063.log 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39837.log.gz 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40267.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40227.log.gz 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35510.log 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.42063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42024.log 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.35510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41829.log 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42063.log.gz 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.42024.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35510.log.gz 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39212.log 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42102.log 2024-03-25T18:28:32.607 INFO:teuthology.orchestra.run.smithi162.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42024.log.gz 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41478.log 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41829.log.gz 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.42102.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39212.log.gz 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41595.log 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42102.log.gz 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39330.log 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41478.log.gz 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39603.log 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41868.log 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41595.log.gz 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39603.log: gzip -5 86.4% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39330.log.gz /var/log/ceph/ceph-client.admin.40462.log 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41868.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29164.log 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39603.log.gz 2024-03-25T18:28:32.608 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40462.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41868.log.gz 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39798.log 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.tmp-client.admin.29164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39954.log 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40462.log.gz 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29164.log.gz 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39681.log 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35333.log 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39798.log.gz 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39681.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41166.log 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39954.log.gz 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.35333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39017.log 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39681.log.gz 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41166.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35333.log.gz 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40698.log 2024-03-25T18:28:32.609 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41166.log.gz 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39017.log.gz 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40503.log 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36411.log 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41946.log 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40698.log.gz 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39095.log 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.36411.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40503.log.gz 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36411.log.gz 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40188.log 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41946.log.gz 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40854.log: gzip -5 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39095.log.gz 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41088.log 2024-03-25T18:28:32.610 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41439.log 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41088.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40188.log.gz 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41127.log 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41088.log.gz/var/log/ceph/ceph-mgr.y.log: 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39251.log 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41439.log.gz 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41712.log 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5/var/log/ceph/ceph-client.admin.39251.log: --verbose -- /var/log/ceph/ceph-client.admin.40815.log 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41127.log.gz 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41712.log: gzip -5 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39251.log.gz 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35030.log 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41712.log.gz 2024-03-25T18:28:32.611 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37577.log 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.35030.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40815.log.gz 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41517.log 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.37577.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35030.log.gz 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41517.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37577.log.gz 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41985.log 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.0.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41517.log.gz 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38978.log 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38939.log 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.38978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40932.log 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41985.log.gz 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.38939.log: 87.2%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38978.log.gz 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40932.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38939.log.gz 2024-03-25T18:28:32.612 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40932.log.gz 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39134.log 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40620.log 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39134.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39134.log.gz 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr: 92.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39642.log 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39993.log 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39642.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40620.log.gz 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42141.log 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39993.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39642.log.gz 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40893.log 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39993.log.gz 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.42141.log: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41400.log 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr: 86.0%/var/log/ceph/ceph-client.admin.40893.log: -- replaced with /var/log/ceph/ceph-client.admin.42141.log.gz 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41322.log 2024-03-25T18:28:32.613 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40893.log.gz 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41322.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41400.log.gz 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41790.log 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41751.log 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41322.log.gz 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39369.log 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41790.log.gz 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39290.log 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40581.log 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41751.log.gz 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.39290.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39369.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41010.log 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40110.log 2024-03-25T18:28:32.614 INFO:teuthology.orchestra.run.smithi162.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39290.log.gz 2024-03-25T18:28:32.615 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.41010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41361.log 86.8% 2024-03-25T18:28:32.615 INFO:teuthology.orchestra.run.smithi162.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40581.log.gz 2024-03-25T18:28:32.615 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:28:32.615 INFO:teuthology.orchestra.run.smithi162.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41010.log.gz 2024-03-25T18:28:32.615 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/ceph-client.admin.40110.log: /var/log/ceph/ceph-client.admin.41361.log: 86.9%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.40110.log.gz 2024-03-25T18:28:32.615 INFO:teuthology.orchestra.run.smithi162.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41361.log.gz 2024-03-25T18:28:32.661 INFO:teuthology.orchestra.run.smithi191.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:28:33.396 INFO:teuthology.orchestra.run.smithi162.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:28:33.409 INFO:teuthology.orchestra.run.smithi162.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:28:33.457 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:28:33.523 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:28:33.667 INFO:teuthology.orchestra.run.smithi162.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:28:33.690 INFO:teuthology.orchestra.run.smithi162.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:28:33.754 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:28:33.794 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:28:37.596 INFO:teuthology.orchestra.run.smithi191.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:28:37.598 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-03-25T18:28:37.598 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m5.031s 2024-03-25T18:28:37.598 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m8.992s 2024-03-25T18:28:37.598 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.324s 2024-03-25T18:28:37.642 INFO:teuthology.orchestra.run.smithi162.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:28:37.910 INFO:teuthology.orchestra.run.smithi162.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:28:37.912 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-03-25T18:28:37.912 INFO:teuthology.orchestra.run.smithi162.stderr:real 0m5.377s 2024-03-25T18:28:37.912 INFO:teuthology.orchestra.run.smithi162.stderr:user 0m13.907s 2024-03-25T18:28:37.912 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m0.520s 2024-03-25T18:28:37.912 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:28:37.913 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/remote/smithi162/log 2024-03-25T18:28:37.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:28:40.406 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/remote/smithi191/log 2024-03-25T18:28:40.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:28:41.956 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:28:42.171 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=deb2b235f65544abb5e8a68396b9737a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:28:42.176 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:28:42.185 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:28:42.228 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:28:42.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:28:42.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:28:42.328 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:28:42.337 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:28:42.337 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:28:42.340 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:28:42.354 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:28:42.355 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:28:42.359 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:28:42.359 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-25T18:28:42.359 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 +827us[ +827us] +/- 67ms 2024-03-25T18:28:42.359 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -1094us[-1102us] +/- 52ms 2024-03-25T18:28:42.359 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +910us[ +902us] +/- 72ms 2024-03-25T18:28:42.360 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:42.361 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:28:42.361 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-03-25T18:28:42.361 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 56 +820us[ +814us] +/- 102ms 2024-03-25T18:28:42.361 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -1068us[-1074us] +/- 52ms 2024-03-25T18:28:42.361 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +917us[ +911us] +/- 72ms 2024-03-25T18:28:42.361 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:42.362 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:28:42.371 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:28:42.372 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:28:42.395 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:28:42.474 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:28:42.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:28:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:28:42.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:28:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:28:42.610 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:28:42.619 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:28:42.748 INFO:teuthology.task.internal:Duration was 8074.927620 seconds 2024-03-25T18:28:42.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:28:42.780 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:28:42.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:28:42.783 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:28:42.829 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:28:42.830 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:28:43.315 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:28:43.315 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-03-25T18:28:43.316 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:28:43.337 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-03-25T18:28:43.337 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:28:43.363 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:28:43.363 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:28:43.381 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:28:43.433 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:28:43.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:28:43.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:28:43.577 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:28:43.587 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:28:43.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:28:43.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:28:43.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:28:43.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:28:43.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:28:43.719 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-03-25T18:28:43.721 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-03-25T18:28:43.758 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:28:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:28:43.812 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:28:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:28:43.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:28:43.837 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:28:43.837 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/remote/smithi162 2024-03-25T18:28:43.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:28:43.888 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254/remote/smithi191 2024-03-25T18:28:43.888 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:28:43.924 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:28:43.925 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:28:43.927 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:28:43.973 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:28:43.986 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:28:43.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:28:44.018 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:28:44.018 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:28:44.021 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:28:44.033 INFO:teuthology.orchestra.run.smithi162.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:28 /home/ubuntu/cephtest 2024-03-25T18:28:44.035 INFO:teuthology.orchestra.run.smithi191.stdout: 265317 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:28 /home/ubuntu/cephtest 2024-03-25T18:28:44.037 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:28:44.067 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:28:44.245 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:28:44.272 DEBUG:teuthology.nuke:shortname: smithi162 2024-03-25T18:28:44.272 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:28:44.297 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:25.553767', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKmDuJIdzytL+sQK2u890CdR23URp81hjZiYk0h3GOYGZrh4ReOTPR/CTG9+Yf3U+ctPKclt7vHrQJeTXjsoR1A='} 2024-03-25T18:28:44.301 DEBUG:teuthology.nuke:shortname: smithi191 2024-03-25T18:28:44.301 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:28:44.324 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:05:25.551908', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP/wg/nRE65EJryN827OMAbsAXfPhKVsgN7uskCi6+GuQK9UkPx1JoN6arIrnJiWmNxcKP2nzDLkXe0PoG+Tv8Q='} 2024-03-25T18:28:44.361 INFO:teuthology.orchestra.console.smithi162:Power off 2024-03-25T18:28:44.362 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:28:44.389 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-03-25T18:28:44.389 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:28:44.405 INFO:teuthology.orchestra.console.smithi191:Power off 2024-03-25T18:28:44.406 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:28:44.415 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-25T18:28:44.429 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2024-03-25T18:28:44.429 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:28:44.452 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-25T18:28:48.417 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:28:48.453 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:28:48.531 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-25T18:28:48.562 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-25T18:28:52.534 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:28:52.563 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:28:52.646 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-25T18:28:52.674 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2024-03-25T18:28:56.648 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:28:56.675 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:28:56.759 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-03-25T18:28:56.786 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2024-03-25T18:28:56.860 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-03-25T18:28:56.887 INFO:teuthology.orchestra.console.smithi191:Power off completed 2024-03-25T18:28:56.992 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 8074.927619934082 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=deb2b235f65544abb5e8a68396b9737a status: fail success: false 2024-03-25T18:28:56.992 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:28:57.073 INFO:teuthology.run:FAIL